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clearly  indicate  that  the  lattice  methods  are  extremely  powerful, 
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ABSTRACT 


The  problem  of  obtaining  parametric  models  for  linear 
and  nonlinear  systems  based  on  observations  of  the  input 
and  output  of  the  system  is  one  of  wide  ranging  interest. 

For  linear  systems,  moving  average  (MA)  and  autoregressive 
CAR)  models  have  received  considerable  attention  and,  based 
on  the  Levinson  algorithm,  a  number  of  very  powerful  methods 
involving  lattice  filter  structures  have  been  developed  to 
obtain  the  model  solutions.  For  nonlinear  systems  the 
Vo], terra  series  model  which  is  a  nonlinear  extension  of  the 
moving  average  model  is  frequently  used. 

The  purpose  of  this  research  is  to  extend  these  tech¬ 
niques  to  more  general  linear  and  nonlinear  models.  Using 
the  equation  error  formulation,  lattice  solution  methods 
in  batch  processing  and  adaptive  form  are  developed  for  both 
single  and  multichannel  autoregressive  moving  average  (ARMA) 
models  for  linear  systems  and  Volterra  series  models  for 
nonlinear  systems.  A  nonlinear  extension  of  the  ARMA  model 
is  also  considered  and  is  shown  in  some  cases  to  remedy 
problems  encountered  in  Volterra  modeling  of  nonlinear  sys¬ 
tems.  Lattice  methods  are  also  developed  for  the  nonlinear 
ARMA  model  and  it  is  shown  that  the  methods  obtained  for 
linear  ARMA  modeling  follow  as  a  special  case  of  the  non¬ 
linear  results,  . . .  . 

Experimental  verification  of  the  methods  developed  for 

i 

single  channel  linear  ARMA  modeling  is  presented  and  used 


to  explore  the  characteristics  of  the  lattice  solution 
techniques.  The  results  clearly  indicate  that  the  lattice 
methods  are  extremely  powerful*  capable  of  producing  highly 
accurate  system  models  using  short  runs  of  data. 
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X .  INTRODUCTION 


Traditionally,  man  has  attempted  to  create  models  of 
portions  of  his  environment  for  two  principal  reasons: 

1.  To  gain  insight  and  understanding  as  to  their 
functioning ; 

2.  As  a  prelude  to  taking  some  action  such  as  attempting 
to  exercise  control  over  them. 

The  field  of  physics  for  instance,  is  replete  with  examples 
where  men  have  created  models  to  study  and  explain  phenomenon 
from  planetary  motion  to  the  motion  and  even  origin  of  sub¬ 
atomic  particles.  In  designing  machines,  engineers  routinely 
rely  on  models  of  the  components  they  use  to  describe  how  they 
will  function,  and  to  obtain  the  desired  results  in  the  final 
product.  Economics  is  another  field  where  the  use  of  models 
abounds  for  such  purposes  as  identifying,  forecasting  or 
trying  to  direct  trends. 

The  scope  of  the  modeling  problem  is  quite  broad  be- 
gining  with  a  decision  on  the  type  of  model  to  be  used,  what 
physical  quantities  to  measure,  how  to  estimate  the  para¬ 
meters  of  the  model  from  the  measurement ,  and  finally  a  veri¬ 
fication  of  the  model.  In  the  chapters  that  follow,  one 
facet  of  this  problem,  that  of  estimating  model  parameters, 
will  be  explored  in  detail  for  a  number  of  linear  and  non¬ 
linear  models. 


A.  THE  PROBLEM  STATEMENT 

The  primary  concern  of  this  work  is  the  determination  of 
discrete  time  models  for  both  linear  and  nonlinear,  time 
invariant  systems  from  sampled  observations  of  the  system 
inputs  and  outputs,  The  general  approach  underlying  all  of 
the  models  examined  here  assumes  that  the  system  to  be 
modeled  is  described  by  the  equation 

y(k)  =  Fi0&i(k-)]  +  F20Cy(k-l)]  +  F30[u(k),y(k-l)]  (1.1) 

where  F1Q,  F2Q  and  F3Q  are  functions  of  past  and  present 
values  of  their  arguments,  and  u(k)  and  y(k)  are  the  system 
input  and  output  respectively.  This  is  depicted  in  Figure 
(1.1).  A  possible  method  for  modeling  this  type  of  system 
is  to  create  a  model  of  exactaly  the  same  configuration  with 
functions  F^q,  F2q  and  assume  a  Form  for  these  functions 

operate  the  system  and  model  in  parallel  with  the  same  input 
and  adjust  the  parameters  of  the  model  functions  to  minimize 

A 

the  mean  square  error  (MSE)  between  the  model  output  y(k) 
and  the  system  output.  The  symbol  ” A"  is  used  here  to  indi¬ 
cate  that  the  signal  is  an  estimate  of  y(k).  This  is  depicted 
in  Figure  1.2  and  is  often  referred  to  as  direct  form  modeling 
since  the  assumed  topology  of  the  system  is  directly  copied 

1  Script  notation  will  be  used  to  refer  to  quantities 
associated  with  the  system  while  nonscript  notation  will  be 
used  for  their  corresponding  approximants  in  the  model. 
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y(k)=F1QCu(k) ]+F2Q[y(k-l)]+?3g[uCk) ,y(k-l)] 


and  the  error  signal  is  referred  to  as  th-i  "output  error". 
As  an  example,  if  the  system  is  linear  an  appropriate  model 


F,nCu(k)]  =  /  j  a(i)u(k-i) 
10  i=0 


(1.3a) 


*  V' 

F_nCy(k-l) ]  =  /Lj  y<k-il 


(1.3b) 


F-nCu(k),y(k-l)]  =  0 


(1.3c) 


(for  linear  models,  F3QC* ]  will  be  zero).  In  general 
however,  the  direct  form  approach  will  have  serious  diffi¬ 
culties  if  either  F 2 Q  C • ]  or  F 3  Q  C • D  are  nonzero  since  the 

/v 

past  values  of  y(k)  used  in  these  functions  are  themselves 
dependent  on  the  model  parameters.  A  mimimum  mean  square 
output  error  approach  results  in  a  system  of  highly  non¬ 
linear  simultaneous  equations  which  must  be  solved  to 
obtain  the  model  parameters. 

To  avoid  these  difficulties,  the  equation  error  approach 
[Refs.  34  and  23]  to  system  modeling  (which  uses  different 
model  forms  in  the  analysis  and  synthesis  phases)  will  be 
applied  to  the  problem.  The  analysis  model  is  depicted  in 
Figure  1.3  and  differs  from  the  direct  form  model  in  that 
Foq  and  F30  are  functions  of  past  and  present  values  of  the 
delayed  system  output  rather  than  the  analysis  model  output. 
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u(k) 


e(k) 


Figure  1.3,  The  equation  error  approach  for 
system  modeling. 

For  each  of  the  models  studied,  a  general  form  for  the  three 
functions  is  assumed  and  the  parameters  of  the  model  (coef¬ 
ficients  of  the  functions)  are  set  to  obtain  a  MMSE  solution. 
In  each  case,  the  MSE  cost  function  is  a  quadratic  function 
of  the  model  parameters  (due  to  both  the  equation  error 
formulation  and  to  the  form  chosen  for  the  functions)  with 
a  unique  minimum  and  therefore  the  solution  is  given  by  a 
system  of  linear  equations.  The  synthesis  model  is  of  the 
same  form  assumed  for  the  system  in  Figure  1.1  and  uses  the 
functions  F10>  F2o  and  ^30  determ;i-ne<:i  during  the  analysis 
phase . 
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mm**** 


As  an  alternative  to  the  topology  shown  in  Figure  1.3 
it  will  occasionally  be  convenient  to  consider  the  error 
signal  e(k)  as  the  output  of  the  analysis  model  rather  than 

A 

the  prediction  y(k)  ,  This  can  he  accomplished  in  one  of  two 
ways  by  defining 

F2Cy(k)]  =  y(k)  -  F20ty(k-D]  (1.4a) 

or 

F3 Cu(k; ,y(k) ]  -  y(k)-F3Q[u(k) ,y (k~l) ]  (1.4b) 

and  reformulating  the  analysis  model  as  shown  in  Figures 

1.4a  or  1.4b.  These  model  forms  are  often  referred  to  as 

prediction  error  models  since  their  outputs  are  the  errors 

in  predicting  y(k)  rather  than  the  predictions  themselves. 

There  are,  however,  no  substantive  differences  between  the 

modeling  approaches  depicted  in  Figure  1.3,  1.4a  and  1.4b. 

The  equation  error  formulation  can  be  generalized  to 

multiple  input  multiple  output  systems  as  well  (henceforth 

referred  to  as  multichannel  systems)  by  considering  u(k)  and 

y(k)  as  vectors  of  input  signals  and  Qq  output  signals 

and  F1Q,  F2Q,  F3Q,  F2  and  F3  as  vector  functions.  The 

prediction  error  signal  e(k)  becomes  a  Q^-vector  of  signals 

and  the  model  parameters  can  be  set  to  minimize  the  trace 

T 

of  the  prediction  error  covariance  matrix  P  =  s{e(k)e(k)  }, 
Such  generalizations  have  been  developed  to  a  degree  in  the 
multichannel  filtering  literature  and  will  be  investigated 
further  here. 
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SYSTEM 


It  is  important  to  keep  in  mind  however,  that  while  the 
equation  error  formulation  can  be  used  to  find  a  model  solu¬ 
tion,  it  is  an  indirect  method  as  opposed  to  the  direct  form 
method  which  minimizes  the  mean  square  value  of  output  error. 
The  direct  form  model  has  been  modified  to  obtain  the  equa¬ 
tion  error  analysis  model  so  that  the  parameters  can  be  ob¬ 
tained  via  the  solution  of  systems  of  linear  equations.  The 
price  paid  for  this  simplification  in  the  model  analysis 
problem  is  that  additive  noise  on  the  measured  system  out¬ 
put  will  introduce  a  bias  in  the  model  coefficient  estimates. 


B.  OVERVIEW 

Chapter  II  along  with  appendices  A  through  F  provide  a 
unified  review  of  the  existing  background  theory  on  minimum 
mean  square  equation  error  modeling  of  linear  systems.  The 
moving  average  (MA)  and  autoregressive  (AR)  models  are  pre¬ 
sented  and  their  relative  merits  compared.  In  Section  II. C. 
the  Levinson  algorithm  [Refs.  9,  10  and  27]  for  the  AR  and 
MA  models  is  developed,  greatly  simplifying  the  solution 
process  for  these  models.  Section  II. D,  then  shows  that  the 
Levinson  algorithm  defines  the  AR  and  MA  models  in  terms  of 
lattice  filter  structures. 

These  lattice  structures  have  received  widespread  atten¬ 
tion  and  have  led  to  a  host  of  new  developments  in  modeling, 
spectral  estimation,  filter  structures  and  adaptive  filtering. 
Examination  of  the  properties  of  these  forms  have  suggested 
a  number  of  new  methods  for  calculating  model  coefficients 


that  offer  increased  accuracy,  and  in  some  cases  guarantee 
model  stability  even  in  the  presence  of  correlation  estimates 
obtained  by  averaging  over  short  time  intervals  [Refs.  5, 

20,  29  and  36].  Applied  by  Burg  [Ref.  5]  to  spectral  esti¬ 
mation,  these  methods  allow  the  determination  of  power 
spectra  via  AR  modeling  from  very  short  runs  of  data  without 
any  need  for  the  use  of  a  window  function.  In  finite  pre¬ 
cision  arithemetic  implementations ,  the  lattice  structures 
have  been  shown  by  Markel  and  Gray  [Ref.  33]  to  be  less  sensi¬ 
tive  to  roundoff  noise  and  coefficient  quantization  than 
direct  structures  and  have  led  to  the  design  of  other 
structures  that  offer  improved  performance  over  conventional 
parallel  realizations.  Griffiths  has  shown  that  these 
lattices  can  be  implemented  adaptively  [Refs.  16,  17  and  18] 
and  that  they  offer  the  potential  for  more  rapid  convergence 
than  conventional  LMS  adaptive  filters.  Recently  Morf  [Refs. 
36,  37  and  38]  has  also  used  these  lattice  structures  to 
implement  a  recursively  updated  deterministic  least  squares 
adaptive  scheme.  It  is  readily  apparent  therefore,  that  thv.. 
original  work  of  Levinson  and  the  lattice  structures  that 
have  evolved  from  it  have  had  an  important  impact  on  the 
field  of  digital  signal  processing, 

In  Section  II. E.  ,  the  multichannel  generalization  of  many 
of  the  single  channel  AR  and  MA  modeling  results  is  presen¬ 
ted.  After  a  discussion  of  the  basic  multichannel  AR  and  MA 
models  [Refs.  26  and  45],  the  multichannel  version  of  the 
Levinson  algorithm  originally  developed  by  Whittle  [Ref.  56], 
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and  Wiggins  and  Robinson  [Ref,  61]  is  presented.  A  new 
form  for  the  models  is  introduced  and  used  here  however, 
to  facilitate  the  application  of  these  results  later  in 
various  other  modeling  problems.  Multichannel  lattice 
structures  are  then  derived  and  from  them  alternative 
solution  methods  for  the  modeling  problems  are  developed. 

Finally,  in  Section  II. F.  the  LMS  adaptive  algorithm 
[Ref.  58]  is  reviewed  and  the  adaptive  implementations  of 
the  lattice  structures  due  to  Griffiths  are  presented. 

In  Chapter  III,  the  more  general  autoregressive  moving 
average  model  is  presented  using  the  equation  error  formu¬ 
lation  attributed  to  Kalman  [Ref.  23].  After  a  brief 
discussion  of  the  model,  new  model  transition  formulas 
are  developed  showing  how  the  ARMA  model  is  related  to  the 
simpler  anu  less  general  AR  and  MA  models.  System  input 
signal  requirements  for  the  ARMA  modeling  process  are  ex¬ 
plored  and  it  is  shown  that  the  power  spectrum  of  the  input 
signal  can  be  considered  as  a  frequency  dependent  weighting 
function  in  the  model  optimization.  Then  the  main  result 
of  the  chapter  is  presented.  With  suitable  assumptions, 
a  recursive  in  order  solution  method  for  ARMA  modeling 
(the  (n+l)=st  order  solution  is  obtained  from  the  n-th 
order  solution)  is  obtained  based  on  the  Levinson  algorithm  for 
multichannel  AR  models.  From  this,  lattice  solution  methods 
for  the  ARMA  model  are  developed  in  both  batch  processing 
and  adaptive  form,  (Batch  processing  here  refers  to  assuming 
ergodicity  and  estimating  correlations  with  time  averaging) . 
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A  similar  result  has  recently  been  presented  by  Morf  [Refs. 

37  and  383  with  the  assumption  of  a  white  noise  input  sig¬ 
nal  to  the  system.  The  results  presented  here  follow  from  a 
different  approach  without  the  assumption  of  a  white  noise 
input.  Experimental  results  are  also  presented  verifying 
the  methods  and  theory,  and  showing  their  advantages  (and 
disadvantages)  over  conventional  ARMA  modeling  methods. 

The  programs  used  in  these  simulations  are  listed  in  Appen¬ 
dix  J.  In  Section  III.F.,  and  Appendix  G,  it  is  shown  that 
these  single  channel  methods  readily  extend  to  the  multi¬ 
channel  ARMA  model,  and  as  one  would  expect,  can  be  obtained 
as  a  special  case. 

In  Chapter  IV  two  types  of  nonlinear  models,  the  Volterra 
series  model  and  the  new  nonlinear  ARMA  model  recently  pro¬ 
posed  by  Parker  [Ref.  64],  are  considered.  After  a  brief 
discussion  of  the  Volterra  model,  it  is  shown  that  the 
solution  can  be  obtained  using  multichannel  MA  lattice  methods 
if  the  regular  form  of  the  Volterra  kernels  is  used  in  place 
of  the  conventional  symmetric  form.  Then  the  nonlinear  ARMA 
model  is  presented  in  Section  IV. B.  and  it  is  shown  that  for 
many  systems ,  this  model  can  remedy  the  problem  of  the  large 
number  of  terms  (ideally  infinite)  required  by  the  Volterra 
model  to  represent  the  system  in  much  the  same  way  that  the 
ARMA  model  solved  the  problem  arising  in  the  MA  model.  In 
Section  IV.B.2  it  is  also  shown  that  by  using  the  regular 
form,  the  solution  for  the  nonlinear  ARMA  model  can  be 
obtained  using  multichannel  AR  lattice  methods  and  that  the 
linear  ARMA  model  solutions  developed  in  Chapter  III  follow 
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as  a  special  case.  Appendix  X  then  presents  two  examples  of 
nonlinear  ARMA  modeling.  First  a  somewhat  academic  example 
of  a  cascade  of  linear  and  nonlinear  subsystems  is  given 
then  a  nonlinear  ARMA  model  is  proposed  for  the  tracking 
behavior  of  a  phase  locked  loop. 

Finally,  in  Chapter  V,  two  applications  for  the  linear 
and  nonlinear  ARMA  modeling  methods  developed  in  Chapters 
III  and  IV  are  discussed  briefly.  (They  are  reduced  order 
modeling  of  complex  systems  and  modeling  for  fault  detection 
and  diagnosis.)  Then  in  Section  V.B.  conclusions  are  drawn 
on  the  results  of  this  work  and  a  list  of  significant  open 
questions  (both  old  unanswered  questions  and  new  ones 
raised  here)  is  compiled. 


II.  DISCRETE  TIME  LINEAR  SYSTEM  MODELING;  BACKGROUND  THEORY 


While  few  physical  systems  are  absolutely  linear,  linear 
models  often  suffice  to  accurately  describe  their  behavior 
under  normal  operating  conditions,  A  rich  body  of  theory 
has  therefore  been  developed  for  the  analysis  and  modeling 
of  linear  systems  [Ref.  22]  and  a  thorough  knowledge  of 
this  theory  is  vitally  important  to  anyone  interested  in 
understanding  the  functioning  of  these  systems.  The  con¬ 
tinuing  expansion  in  the  availability  of  powerful,  inexpen¬ 
sive  digital  computing  capabilities  has  also  made  discrete 
time  techniques  take  on  a  special  prominence.  With  this 
as  motivation,  the  portion  of  the  background  theory  in 
discrete  time  linear  modeling  upon  which  much  of  the  re¬ 
mainder  of  this  work  depends,  is  developed  here  from  the 
unifying  standpoint  of  a  minimum  mean  square  equation  error 
model  solution. 

The  moving  average  and  the  autoregressive  models  are 
developed  first  for  single  input  single  output  systems. 

Their  solution  via  the  Levinson  algorithm  is  presented  and 
from  this  algorithm  alternate  solution  methods  based  on 
lattice  filter  structures  are  derived.  It  is  shown  that 
almost  all  of  these  results  can  be  generalized  to  the 
multiple  input  multiple  output  case  and  the  corresponding 
multichannel  modeling  methods  are  developed.  Finally, 
adaptive  implementation  of  the  modeling  methods  for  both 


the  conventional  filter  structures  and  the  lattice  filter 
structures  is  presented  as  an  alternative  means  of  obtaining 
model  solutions . 

A.  MOVING  AVERAGE  MODELS 

The  moving  average  (MA)  model  was  among  the  earliest 
discrete  models  developed,  [Refs.  4,  11  and  19]  It  estimates 
the  current  value  of  the  output  of  a  system  as  a  weighted 
combination  of  the  present  value  and  N  past  values  of  the 
system  input  where  N  is  the  order  of  the  model.  The  problem 
then  is  to  estimate  the  weighting  function  or  impulse  re¬ 
sponse  of  the  MA  model  in  some  fashion.  Since  the  MA  model 
characterizes  a  system  in  terms  of  a  finite  duration  approx¬ 
imation  of  its  impulse  response  and  since  any  linear  time 
invariant,  single  input  single  output  system  is  completely 
specified  by  its  impulse  response,  the  MA  model  is  quite 
general  and  can  be  used  for  a  wide  class  of  systems.  De¬ 
fining  (N+l) -vectors  of  model  weights  and  input  data  as 

T 

a+  »  [a( 0)  • • •  a(N) ]  1,2  (2.1a) 


A  superscript  "+"  is  used  to  indicate  that  in  spite  of 
the  fact  that  these  vectors  are  used  for  a  N-th  order  model, 
they  are  (N+l)-vectors  with  elements f indexed  from  zero  to  N 
rather  than  from  one  to  N.  Superscript  T  demotes  transpose. 

2 

Superscripts  in  parenthesis  will  later  be  added  to  the 
model  coefficient  vectors  to  explicitly  indicate  their  de¬ 
pendence  on  the  order  of  the  problem  being  solved.  They  are 
omitted  for  simplicity  however  whenever  doing  so  does  not 
result  in  ambiguous  notation. 
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[u(k)  • • *  u(k-N) ] 


(2.1b) 


u+(K) 


T 


the  MA  estimate  of  the  system  output  becomes 


y(k) 


2  a(n)u<k-n)  =  u+(k)  a+ 
n=0 


In  terms  of  the  modeling  approach  of  Figure  1.3,  F2Q  and 
F30  are  assumed  to  be  zero.  F^  is  a  linear  time  invariant 
function  of  past  and  present  values  of  u(k) .  Assuming  sta- 
tionarity,  an  expression  for  the  mean  square  value  of  the 
error  as  a  quadratic  function  of  the  weights  (a(n)}  is 
given  by 

E,  *  a+TR  +  -  2  a+  r  +  +  ECO )  C2.'2) 

u  u  u  y 

where  in  general  R  .  (n)  =  e  {v(k)w(k+n) } ,  r  ,  =  e  (v(k)w(k+n) }  , 

VW  —•VW  — 

T 

Rvw  5  e{v(k)w(k)  }  and  e{  }  denotes  expectation. 


£  +  ♦ 
u  u 


V(0)  •••  Ruu(-N) 


Euu(N)  •••  Ruu(0) 


E  +  =  [Ruy<°>  V<N)] 


u  y 


uy 


The  surface  described  by  equation  2.2  can  be  pictured  as  a 
concave  hyperparaboloid  with  a  unique  minimum  and  the 
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characteristics  of  such  a  surface  are  described  in  Appendix 
F.  For  example  when  N=l,  the  MSE  as  a  function  of  a(Q)  and 
a(l)  appears  as  shown  in  Figure  2.1. 


a(l) 

/ 


Figure  2.1.  MSE  as  a  function  of  model  weights 
for  a  first  order  (N=l)  MA  model. 


The  minimum  mean  square  error  solution  for  the  coeffi¬ 


cients  is  given  by 


and  the  corresponding  minimum  value  of  the  cost  function 
E2  is 


Ryy<0)  '  SOPT  £u+y 


(2.4) 


Equation  (2.3)  is  a  discrete  time  matrix  form  of  the  Wiener 
Hopf  equation 


Ruy(t)  5  /  RuuCt-X)  hU)  dX 


(2,5) 


where  u(t)  and  y(t)  are  the  continuous  input  and  output 
signals  and  h(t)  is  the  system  impulse  response.  The  pro- 
cess  of  finding  aJpT  in  equation  (2.3) is  the  discrete  time 
equivalent  of  deconvolving  the  input  autocorrelation  func¬ 
tion  from  the  cross  correlation  of  input  and  output  to 
obtain  the  system  impulse  response  in  equation  (2.5). 
Consequently  the  MA  modeling  process  has  been  called  dis¬ 
crete  Wiener  filtering  or  stochastic  deconvolution. 

This  model  constitutes  a  direct  form  approach  as  defined 
in  section  1.1  but  does  not  encounter  difficulty  in  obtaining 
the  model  weights  since  both  F2q  and  are  assumed  to  be 
zero.  As  such,  it  possesses  the  advantage  that  the  estimates 
of  the  model  parameters  will  not  be  biased  by  the  presence  of 
additive  noise  on  the  output  of  the  system  as  shown  in 
Figure  2.2,  as  long  as  the  noise  is  uncorrelated  with  the 
input  signal.  This  can  readily  be  seen  by  replacing  v  in 
equation  2.3  by  y  +  v.  Additive  noise  on  the  input  signal 


v(k) 


Figure  2.2.  Moving  Average  Modeling 

however  will  adversely  affect  the  modeling  process,  and 
introduce  a  bias  in  the  solution  for  the  model  coefficients. 

In  the  transform  domain,  the  model  can  he  represented  by 
a  polynomial  in  powers  of  z~^  and  has  therefore  been  referred 
to  as  an  all  2ero  model 

N 

ACz)  =  £  a(n)  z“n  (2.6) 

n=0 

In  terms  of  this  transfer  function  relationship,  any  bias 
introduced  in  one  or  more  of  the  model  coefficients  has  the 
effect  of  shifting  the  zero  locations  of  the  model. 

In  summary  a  discussion  of  the  advantages  and  disad¬ 
vantages  of  MA  modeling  is  instructive. 

Advantages : 

1)  The  solution  for  the  model  parameters  involves 

i 

only  linear  equations . 
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2) 


The  solution  is  unbiased  in  the  presence  of 
additive  noise  on  the  system  output  as  long  as 
the  noise  and  system  input  are  uncorrelated. 

3)  Since  the  model  is  nonrecursive  it  is  always 
stable. 

Disadvantages : 

1)  The  number  of  terms  (N+l)  needed  for  sufficient 
model  accuracy  may  be  quite  large. 

2)  The  solution  of  a  large  system  of  equations  is 
required . 

3)  The  required  correlation  terms  are  usually  not 
known  and  must  be  estimated  by  assuming  ergodi- 
city  and  averaging  in  time.  This  requires  the 
data  to  be  windowed  and  set  to  zero  outside  the 
averaging  interval  in  order  to  maintain  the  even 
symmetry  of  the  autocorrelation  functions . 

4)  The  modeling  process  is  restricted  to  linear 
time  invariant  systems . 

B.  AUTOREGRESSIVE  MODELS 

The  autoregressive  (AR)  model  attempts  to  deal  with  one 
of  the  difficulties  (1)  encountered  in  MA  modeling;  the  need 
for  a  large  number  of  coefficients  to  accurately  describe 
the  model.  [Refs,  2,  4,  11,  19  and  28]  In  AR  modeling, 
which  is  si  letimes  referred  to  as  linear  prediction,  a  pre¬ 
diction  error  approach  is  considered  where 

N 

e(k)  =  y(k)  -  £  Mn)  y(k-n)  (2.7a) 

n=l 
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This  can  be  written  as 

e(k)  =  y(k)  -  y(k) 

=  y(k)  -  £(k)Tb  (2.7b) 

y_(k)  =  Cy(k«l)  •••  y(k-N)]^  (2.7c) 

b  =  C  b ( 1 )  •••  b(N) ]T  ( 2 . 7d) 

Here  F1Q  and  F3Q  are  assumed  to  be  zero  (this  assumption 
will  be  modified  later  to  allow  a  dependence  on  the  input 
signal  in  the  synthesis  phase)  and  F2Q  provides  an  estimate 
of  the  current  value  of  the  system  output  as  a  weighted  sum 
of  N  past  outputs.  The  mean  square  value  of  prediction 
error  as  a  quadratic  function  of  the  weignts  (b(n)}  is 
given  by 

E2  ■  -  2  bTryy  ♦  Ryy(0)  (2.8a) 

and  the  corresponding  MMSE  solution  for  the  weights  is 
given  by 


with 


and 


-yy^OPT  =  -yy 


(2.8b) 


28  ’ 


with 


Using  equation  2.7a,  an  expression  can  be  written 
in  the  transform  domain  for  the  prediction  error  model  which 
accepts  y(k)  as  its  input  and  produces  the  error  sequence 
e(k)  as  its  output. 


Y~(V)-*  =  1  -  2  b(n)z  n  *  B(z)  (2.9) 

n=l 

If  it  is  assumed  that  the  system  input  output  relationship 
can  be  represented  in  transfer  function  form  with 


Y(z )  _ 

UTzT  ‘ 


H(z) 


<x(0) 


a(0) 

BTzT 


(2.10) 


and  that  the  model  parameters  can  be  determined  so  that  B(z) 
=  8(z),  then  the  prediction  error  output  will  be  exactly 
e(k)  =  a(0)  u(k).  For  this  reason  AR  prediction  error 
modeling  has  often  been  called  inverse  filtering  since  the 
prediction  error  filter  essentially  reverses  the  actions  of 
the  system  (with  the  exception  of  a  gain) .  Since  the 
analysis  model  is  in  this  inverse  form  rather  than  in  the 
direct  form,  the  presence  of  additive  noise  on  the  measure¬ 
ment  of  the  system  output  as  shown  in  Figure  2.3  will  intro¬ 
duce  a  bias  in  the  solution  for  the  model  parameters.  This 
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Figure  2.3.  Autoregressive  prediction  error 
modeling  as  an  inverse  filtering 
process . 


has  the  effect  of  shifting  the  roots  of  B(z)  which  are  es¬ 
timates  of  the  poles  of  the  system  and  is  the  price  paid 
for  the  ability  to  obtain  the  model  solution  from  a  set  of 
linear  equations. 

Thus  far,  only  the  analysis  portion  of  the  AR  modeling 
process  has  been  discussed.  With  the  inverse  filtering 
interpretation  of  the  prediction  error  analysis  model,  a 
reasonable  synthesis  model  is  given  in  transfer  function 
form  as 

HU>  =  §£§}  (2.U) 


with  the  gain  term  set  so  that  the  mean  square  value  of 
a(Q)  uCk)  is  the  same  as  that  of  the  prediction  error 
signal.  Thus  it  follows  that 


a(0) 2  = 


e{e(k)2} 


(2.12) 
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Since  the  synthesis  model  is  in  the  form  of  an  all  pole 
filter,  an  appropriate  impulse  response  with  infinite  dura¬ 
tion  might  be  obtained  using  a  low  order  model  (small  N) , 
a  result  that  is  impossible  to  obtain  in  any  finite  order  HA 
model.  This  is  not  to  say  however  that  a  low  cr  even  finite 
order  AR  model  will  always  be  an  appropriate  model  for  any 
linear  system.  If  the  transfer  function  representation  for 
a  system  contains  both  zeros  and  poles,  no  finite  order  AR 
or  MA  model  can  serve  to  exactly  represent  it.  This  fact 
can  be  understood  by  considering  the  form  of  a  geometric 
series 

- ~T  3  E  (Cz*1)11  for  I  Cz**1 1  <1  (2.13) 

1-Cz  1  n=0 

which  shows  that  a  single  pole  can  be  represented  by  an 
infinite  number  of  zeros  and  visa  versa.  Thus  if  the  sys¬ 
tem  has  a  single  zero,  a  high  order  AR  model  may  be  required 
to  represent  it  with  sufficient  accuracy. 

In  summary,  the  advantages  and  disadvantages  of  AR 
modeling  may  be  listed  as  follows: 

Advantages : 

1)  The  solution  for  the  model  parameters  involves 
only  linear  equations , 

2)  Sometimes  an  appropriate  infinite  impulse  re¬ 
sponse  can  be  obtained  with  a  small  number  of 
parameters  in  the  model. 

3)  Direct  knowledge  or  measurement  of  the  system 
input  is  not  required  for  determing  the  system 
poles .  Only  a  knowledge  of  its  mean  square 


value  is  necessary  for  determining  the  gain 
factor . 

Disadvantages : 

1)  The  model  is  biased  by  the  presence  *  -^ditive 
noise  on  the  measured  system  output  signal. 

2)  The  number  of  terms  required  for  sufficient 
model  accuracy  may  be  quite  large  if  zeros  are 
present  in  the  system.  If  this  occurs,  the 
inversion  of  a  large  matrix  will  be  required. 

3)  The  required  correlation  terms  are  usually  not 
known  and  must  be  estimated  by  assuming'  ergodicity 
and  using  time  averages. 

4)  The  modeling  process  is  restricted  to  linear 
time  invariant  systems. 

This  list  of  advantages  and  disadvantages  is  quite 
similar  to  the  one  compiled  for  MA  models  with  two  notable 
differences;  the  bias  in  the  model  and  the  absence  of  a 
requirement  for  input  measurements .  This  second  point  is 
significant  in  that  it  has  led  to  the  application  of  AR 
modeling  to  many  problems  where  an  input  signal  is  unmea¬ 
surable  or  indeed  does  not  exist  including  speech  modeling 
and  spectral  estimation.  [Refs,  2,  5,  12,  15,  21,  32  and  44] 
The  noise  problem  has  restricted  the  process  to  applications 
where  measurements  with  sufficiently  high  signal  to  noise 
ratio  are  available,  making  the  effects  of  the  bias  minimal. 
[Refs,  24  and  43] 
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C.  RECURSIVE  IN  ORDER  SOLUTIONS  FOR  AR  AND  HA  MODELS 


The  preceeding  disucssions  of  the  AR  and  MA  modeling 
problems  tacitly  as summed  an  a  priori  knowledge  of  the  correct 
model  order.  If  this  knowledge  is  not  available  a  reasonable 
approach  for  determining  the  correct  model  order  must  be 
developed  [Ref.  53],  A  commonly  employed  strategy  is  to 
successively  increment  the  model  order  while  observing  the 
MSE  until  further  increases  fail  to  substantially  reduce  the 
MSE.  This  requires  solving  for  a  number  of  different  models 
and  earn  be  an  arduous  task  if  equations  (2.8b)  or  (2.3)  are 
employed  directly. 

The  autocorrelation  matrices  appearing  in  the  AR  and 
MA  model  equations  (2.8b)  and  (2.3)  are  highly  structured 
matrices  (both  Toeplitz  and  symmetric)  and  this  fact  can  be 
exploited  to  facilitate  the  solution  of  these  equations. 

The  Levinson  algorithm  [Refs.  9,  10  and  27]  makes  use  of 
this  structure  to  obtain  model  solutions  recursively  in 
order,  that  is,  the  solution  for  the  n-th  order  model  is 
assumed  to  be  known  and  the  solution  for  the  (n+l)-st  order 
model  is  then  obtained  from  it.  In  this  manner  it  is  pos¬ 
sible  to  start  with  a  first  order  AR  or  a  zeroth  order  MA 
solution  given  by  a  single  equation  and  build  up  the 
desired  order  solution.  The  AR  model  will  be  treated  first 
since  it  is  a  special  case  that  simplifies  the  analysis. 

The  simplifications  arise  due  to  the  fact  that  the  r  vector 
on  the  right  hand  side  of  equation  (2.8b)  is  made  up  pri¬ 
marily  of  terms  also  appearing  on  the  left  hand  side  in  R 
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Superscripts  in  parenthesis  are  used  to  explicitly  indicate 
the  order  of  the  problem  when  specifically  needed. 

1 .  The  Levinson  Algorithm  For  AR  Modeling 

The  n-th  order  AR  model  solution  of  equation  2.8b 
is  given  by 


0  (n)  .  (n)  _  _  (n) 

-yy  *  -  -yy 


(2.14) 


The  Levinson  algorithm  assumes  a  relationship  between  the 
n-th  and  (n+l)-st  order  solutions  given  by 


.  (n+1)  _ 

b(n) 

£(n) 

b  = 

0 

••  tm 

k(n+l) 

(2.15) 


and  solves  for  the  vector  and  the  coefficient  k^n+1^. 
Define  permuted  versions  of  the  vectors  b^n^  and  r  as 
f  and  fi.yyn  by  reversing  the  order  of  their  elements. 


b(n)(n) 

b(l)Cn) 


(2.16) 


(1) 


Because  of  the  Toeplitz  symmetric  structure  of  the  auto¬ 
correlation  matrix,  equation  (2,14)  can  also  be  written  as 


(2.17) 
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and  this  relationship  is  essential  in  the  development  of  the 
Levinson  algorithm,  (To  apply  the  algorithm  therefore  when 
time  averaged  estimates  of  the  needed  correlations  are  used, 
the  data  must  be  windowed  prior  to  averaging  to  maintain  the 
even  symmetry  in  the  autocorrelation  function  estimates 
produce  the  required  structure  in  the  autocorrelation  matrix.) 
Making  use  of  equation  (2.15),  in  the  (n+l)-st  order  version 
of  equation  (2.14),  and  using  the  relationship  of  (2.17)  to 
solve  for  and  k^n+^  results  in 


(n)  _  -(n).  (n+1) 

e  —  -r  k 


(2.18a) 


,(n+l) 


R  (n+1)  -  p(n)  b(n) 


R  (0)  -  b(n)  r  (n) 

yy  -  -yy 


(2.18b) 


Therefore,  in  using  equations  2,15  and  2.18  to  obtain 
(n) 

from  b  via  the  Levinson  algorithm  only  one  new  piece  of 
information,  k^n+1\  need  be  calculated.  The  denominator 
of  equation  (2.18b)  can  also  be  recognized  from  equation 
(2,8c)  as  the  MMSE  for  the  n-th  order  AR  model  E2^n\  and 
thus  there  is  little  concern  over  the  possibility  of  it  being 
zero.  If  the  n-th  order  solution  produces  a  perfect  predic¬ 
tion  (zero  MSE)  there  is  no  point  in  trying  to  find  a  better 
prediction  by  increasing  the  order  to  n+1.  The  evaluation  of 
equation  (2.18b)  can  be  further  simplified  by  observing  that 
the  MSE  also  follows  a  recursion  from  one  order  to  the  next 
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given  by 


(n+1) 

2 


(n) 

2 


(n+1) 


2 

] 


(2.19) 


making  it  unnecessary  to  evaluate  the  denominator  at  each 
value  of  n,  (Details  of  this  derivation  are  omitted  here 
but  included  in  Appendix  A  in  the  derivation  of  the  more 
general  multichannel  Levinson  algorithm.)  This  relation  for 
the  propagation  of  mean  square  prediction  error  also  leads 
to  the  restriction  that  k^n+^  must  be  bounded  in  magnitude 
by  unity. 

2 .  The  Levinson  Algorithm  For  MA  Modeling 

Next  consider  the  n-th  order  MA  model  given  by 


R  <n)  +  (n)  .  ^  (n) 

5.  +  +  §.  *  £  + 

u  u  u  y 


(2.20) 


and  again,  assume  a  relationship  between  the  (n+l)~st  order 
and  n-th  order  solutions  given  by 


(2.21) 


+  ( n+ 1 ) 

+  (n) 
a 

yCn+l) 

a  = 

0 

+ 

.g(ntl>. 

/  ~  \ 

Notice  that  in  this  n-th  order  problem,  R  +  +  is  actually  a 

u  u 

n+1  by  n+1  matrix  and  could  be  written  as  Ru^n+1^.  Define 
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Using  equations  (2,21)  and  (2,22)  in  the  n+1  order  MA  model 
equation  it  follows  that 


y(n+l)  .  ^  f(n+l)g(n+l) 


(2.23a) 


where  f^n+^  is  defined  in  a  manner  similar  to  (2.16)  and  is 
comprised  of  the  coefficients  that  arise  in  an  (n+l)-st 
order  autoregression  on  the  input  signal  u(k) , 

Therefore  to  obtain  a  moving  average  model  relating  the 
system  output  signal  y(k)  to  the  input  signal  u(k) ,  an 
autoregressive  model  for  the  system  input  must  first  be 
solved.  Furthermore, 


.(n+1) 


R  (n+1)  n  <n+l)T  +(n) 

Ruy<rv fl)"£uu  & 


R  .  (Q)~p  (n+1)  f(n+1) 
uu  --uu  - 


(2.23b) 


and  the  denominator  of  equation  (2.23b)  is  the  MMSE  in  the 
(n+l)-st  order  autoregression  on  the  signal  u(k) . 

It  is  significant  to  note  that  in  applying  the 
Levinson  algorithm  to  find  a  given  order  AR  or  MA  model,  all 
lower  order  models  along  with  their  MSE's  are  obtained. 

Also,  intermediate  quantities  emerge  (the{k^n;>  in  the  AR 
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model  and  the  {k^n^}  and  {g^}  in  the  MA  model)  which  fully 
characterize  the  models  and  could  be  used  as  an  alternative 
to  the  (a(n)}  or  (b(n)}  coefficients.  This  point  will  be 
developed  further  in  subsequent  sections. 


D.  LATTICE  FORM  AR  AND  MA  MODELS 

The  Levinson  algorithm  derived  in  the  previous  section 
can  be  used  to  derive  lattice  structures  to  implement  the 
MA  model  and  the  AR  analysis  and  synthesis  models  as  alter¬ 
natives  to  a  tapped  delay  line  type  of  implementation  using 
the  coefficients  a(n)  or  b(n)  directly.  [Refs.  29,  30,  32 
and  33] 

1 ,  The  AR  Modeling  Lattice  Structures 

From  the  relationship  between  the  (n+l)-st  and  n-th 
order  solutions  to  the  AR  modeling  problem  determined  in 
equations  (2.15)  and  (2.18a)  it  follows  that  the  transfer 
function  of  the  prediction  error  model  can  be  written  re¬ 
cursively  in  order  as 

jj(n+D(z)  .  B(nJz)  ,  k(n+1)z"n~1B(n)  (z"1)  (2.24 


Defining  a  new  transfer  function 


B(n)(z)  =  z-nBln\z-h 


(2.25a) 


equation  (2.24)  can  be  written  as 


B(n+1)(z)  =  B(n)(z)  -  k(n+1)z-1I(n) (z) 


(2.25b) 
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(z)  recur- 


and  an  expression  can  also  be  written  for  S  n  ^ 
sively  in  order  by  rewriting  equation  (2.25a)  for  order 
(n+1)  and  substituting  equation  (2,24)  yielding, 

S(n+1)(z)  =  -  k<nH-1!B<n)(2)  (2.250 

As  discussed  earlier  in  connection  with  equation  (2.9), 
B^n\z)  describes  the  n-th  order  prediction  error  model  and 
when  its  input  is  the  system  output  Y(z),  it  produces  the 
n-th  order  prediction  error  signal. 

E(n)(z)  =  B(n)(z)  YCs)  (2.26) 

v.  In  the  time  domain  this  signal  can  be  interpreted  as  the 

error  in  predicting  y(k)  forward  in  time  from  a 
weighted  combination  of  the  n  past  values  {y (k-1) * • ’y(k-n) } . 
To  understand  the  significance  of  fl^n^(z)  consider  the  out¬ 
put  signal  when  this  model  is  excited  by  Y(z). 

E(n)(z)  =  B(n)(z)Y(z) 

=  z"nU  -  t  E<n)(i)z+ittCO  (2.27) 

Lai 

In  the  time  domain,  e^n^(k)  can  be  interpreted  as  the  error 

in  predicting  y(k-n)  backward  in  time  from  a  weighted 

«  *  •  ombination  of  the  future  signals  {y(k-n+D  •  •  *y(k) }  ,  These 

•*  * 

n-th  order  forward  and  backward  prediction  processes  at  time 
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kQ  are  illustrated  m  Figure  2.4.  Henceforth,  an  overbar 
will  always  be  used  to  denote  quantaties  associated  with 
backward  in  time  predictions. 


(k>  f 


Forward 

Prediction 

Backward 

Prediction 


Figure  2.4.  Forward  and  Backward  Prediction  Error 
Filtering . 


From  equations  (2.25b)  and  (2.25c)  equations  can  be  written 
recursively  in  order  for  these  forward  and  backward  predic¬ 
tion  error  sequences  as : 


etn+1)<k>  »  e(n)(k)  -  k(n+1)S<n)Ck-l> 


-(n+i)(k;  .  r<n)(k  ,,  .  k(n+l)  (n)(k) 


(2.28a) 


(2.28b) 


[i] 


Noting  that  the  prediction  error  for  a  zero-order  AR  pre¬ 
dictor  of  y(k)  (or  no  predictor  at  all)  is  just  the  signal 
y(k)  itself, 

e(o)(k)  =  e(o)(k)  =  y(k)  (2.28c) 

the  prediction  error  filter  can  be  drawn  in  lattice  form  as 
shown  in  Figure  2.5  for  a  second  order  model, 


Figure  2.5.  Lattice  Form  Of  A  Second  Order  Pre¬ 
diction  Error  Model. 

This  structure  has  many  interesting  properties , 
among  the  most  important  of  which  is  the  successive  de¬ 
coupling  property.  In  going  from  one  order  AR  model  to  the 
next,  all  of  the  previously  determined  transfer  function 
coefficients  {b(n)>  will  generally  change.  The  Levinson 
algorithm  shows  however  that  only  one  new  piece  of  infor¬ 
mation  is  needed  to  obtain  the  optimum  (n+l)-st  order 
solution  from  the  optimum,  n-th  order  solution  (see  equation 
(2.24)),  In  terms  of  the  lattice  filter  of  Figure  2.4  this 
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means  that  given  the  optimum  n-th  order  model  in  lattice 
form,  one  need  only  add  another  stage  to  the  structure, 
setting  the  coefficient  of  that  stage  k^n+^  to  minimize 
the  mean  square  value  of  e^n+1^(k).  Nothing  in  the  first 
n  stages  need  be  changed.  The  overall  high  order  minimi¬ 
zation  problem  is  in  this  fashion  decomposed  into  a  se¬ 
quence  of  first  order  minimizations,  one  at  each  lattice 
stage. 


% 


Another  important  property  of  the  lattice  which  can 
be  proven  and  will  be  of  use  later  is  the  orthonogalization 
of  the  backward  prediction  error  sequence  [Ref.  32]  which 
states  that 


e{e(i)(k)e<j)(k)> 


0  i*j 


*r  (i) 
L2 


1  =  3 


(2.29) 


Thus  it  is  seen  that  a  set  of  orthogonal  signals  (the  back¬ 
ward  prediction  errors  at  the  various  stages)  are  generated 
as  a  by-product  of  the  lattice  model. 

As  a  consequence  of  the  successive  decoupling  pro¬ 
perty  of  the  lattice,  a  number  of  alternatives  to  equation 
(2,18b)  for  determining  the  lattice  coefficients  can  be 
found.  The  most  obvious  method  is  to  set  k^n+^  to 
explicitly  minimize  the  mean  square  value  of  forward  pre¬ 
diction  error  in  equation  (2,28a)  at  the  (n+l)-st  order 
stage  given  the  best  lattice  of  order  n.  This  is  termed 
the  forward  method  and  is  denoted  by  a  subscript  F  on  the 
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and  since  they  are  both  driven  by  the  same  input,  Y(z),  the 
mean  square  values  of  both  the  forward  and  backward  predic¬ 
tion  error  signals  at  a  given  stage  are  the  same  making 
equations  (2.30)  and  (2.31)  equivalent.  It  is  a"1  so  possible 
to  show  that  they  are  equivalent  to  equation  (2.18b). 
Recognizing  that  the  required  expectations  will  eventually 
have  to  be  estimated  by  using  time  averages,  these  two 
methods  for  calculating  k^n+^  will  not  in  general  be 
exactly  equivalent  and  it  might  be  preferable  to  use  the 
arithmetic  mean  of  the  mean  square  values  of  forward  and 
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r  jam  ^^BK^mi^mmKmamK, 


backward  prediction  error  as  a  cost  function 


>s[e{e(n+1)(k)2  +  £(n+1)(k)2}] 


(2.33a) 


This  leads  to  a  third  method  derived  by  Burg  [Ref.  5]  in 
his  work  on  maximum  entropy  spectral  analysis  and  given  by 


.(n+1)  2s{e(n?(k)e(n)  (k-1)  >  _ 

^BG  "  e{e(n)(k)2}+e{e(n)(k-l)2} 


(2.33b) 


Notice  that  is  the  harmonic  mean  of  kpn+1^  and  kgn+1^. 
A  fourth  method  due  to  Itakaura  and  Saito  [Ref.  20]  can  also 
be  derived  which  results  in 


.(n+1)  _  e{e(n)(k)e(n)(k-l)> 


"IS 


V  6  {e^  (k) 2 } e  (iCn)  (k-1) 2 } 


(2.34) 


and  is  simply  the  geometric  mean  of  the  forward  and 
backward  coefficients. 

Since  equation  (2.34)  is  of  the  form  of  a  normalized 
correlation  k^g  will  always  be  bounded  by  unity  in  magnitude 
as  required  by  equation  (2,19).  Furthermore  since 


j  Harmonic  Mean  j  _<  |  Geometric  Mean  j 

it  follows  that  kg^+1^  will  be  similarly  bounded.  These 
bounds  are  significant  since  Markel  [Ref.  32]  has  shown  that 
1 k ^ n ^ | <1  is  a  necessary  and  sufficient  condition  to  ensure 


44 


that  the  roots  of  Bv  '(z)  be  within  the  unit  circle  guaran¬ 
teeing  the  stability  of  the  n-th  order  all  pole  model.  No 
such  guarantees  of  model  stability  exist  when  the  forward 
or  backward  solution  methods  of  equations  (2.30)  and  (2.31) 
are  used  with  the  correlation  estimates  obtained  by  aver¬ 
aging  for  finite  time  intervals, 

To  determine  the  AR  synthesis  model  in  lattice  form 
it  is  only  necessary  to  rewrite  equation  (2.28a)  as 

e<n)(k)  =  e<n+1)(k)  +  k(n+1)'e(n)  (k-1)  (2.35) 

Together  with  equations  (2.28b)  and  (2.28c),  this  describes 
the  structure  shown  in  Figure  2,6  for  a  second  order  case 
and  when  it  is  driven  by  the  second  order  prediction  error 
signal,  it  will  reconstruct  y(k)  exactly.  Thus  it  imple¬ 
ments  the  transfer  function  —  ^ —  or,  in  general,  when 

1  8  ( z ) 

stages  are  used  '~rrnt - .  .  Recognizing  that  if  the  pre-  . 

B;cu(z) 

diction  error  model  is  an  accurate  model  of  the  system 
denominator  polynomial,  e^  (k)  =a(o)u(k) ,  this  input  signal 
is  used  in  the  synthesis  model.  Because  of  analogies  with 
transmission  lines  and  wave  propagation  models,  the  lattice 
coefficients  have  been  referred  to  as  reflection  coefficients. 
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Figure  2.6.  Lattice  Form  Of  The  All  Pole  Synthesis 
Model  For  The  Second  Order  Case. 


2 .  The  MA  Modeling  Lattice  Structure 

A  similar  lattice  form  is  applicable  to  the  MA 
modeling  problem.  From  equations  (2,21)  and  (2.23a)  the 
transfer  function  of  the  MA  model  can  be  written  recursively 
in  order  as 


A<n+1>Cz)  = 


A(n)(z) 


gCn+l)g6*lu, 


(2.36) 


where,  as  discussed  in  connection  with  equation  (2,23a), 

nli 

S0--A(z)  is  the  backward  prediction  error  transfer  function 
for  an  autoregressive  model  of  the  input  signal  u(k) . 
Multiplying  both  sides  of  equation  (2.36)  by  U(z)  and  trans¬ 
forming  into  the  time  domain  it  follows  that 


r.(n+l) 


(k)  = 


;(n\k> 


g(n*l)-n*lCk) 


(2.37) 
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where  en+1(k)  is  the  backward  prediction  error  signal  from 
the  autoregression  on  the  input  signal  u(k) ,  and  can  he 
obtained  by  operating  a  prediction  error  lattice  with  u(k) 
as  its  input.  Then  with  the  additional  term  in  equation 
(2.37)  the  lattice  form  of  the  MA  model  can  be  drawn  as 
shown  in  Figure  2.7. 


Figure  2,7.  Lattice  Form  Of  The  MA  Model  (Second 
Order  Case) . 


It  was  stated  earlier  that  the  AR  prediction  error 
lattice,  as  a  by-product,  forms  a  set  of  orthogonal  or 
uncorrelated  backward  prediction  error  signals  from  its 
input.  Here  in  the  MA  model,  these  orthogonal  signals  are 
linearly  combined  to  form  the  HA  estimate  of  the  system 
output.  If  the  input  signal  u(k)  is  a  white  process,  an 
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examination  of  any  of  the  solution  methods  previously  dis¬ 
cussed  will  show  that  all  of  the  {k^n^}  lattice  coefficients 
will  be  zero  since  the  delayed  samples  of  a  while  process 
are  already  orthogonal.  Otherwise  the  {k^}  lattice 
coefficients  will  be  set  to  orthogonalize  the  backward  pre¬ 
diction  error  signals.  As  a  consequence  of  this,  the 

weighting  coefficients  gv  can  be  set  independently  of 

/  n  \ 

each  other1,  that  is  g  can  be  set  to  minimize 


e{e£n)(k)2}  =  e{[y(k)  -  y(n!k)]2}  (2.38) 


*  ( n) 

given  the  best  prediction  of  order  n-l,y  (k) .  This  results 
in  an  alternative  expression  to  equation  (2.23b)  for  g^ 
given  by 


(n)  .  e(n)(k)>  _  g  (y(k)  e(n?(k)> 

£  {V^Yk)  e{Vn)(k>2} 


(2.39) 


Here  e£n\k)  is  the  error  between  the  system  output  and  its 
n-th  order  MA  estimate. 


E.  MULTICHANNEL  AR  AND  MA  MODELING 

Both  the  AR  and  MA  modeling  problems  previously  dis¬ 
cussed,  as  well  as  their  solution  via  the  Levinson  recursion 
and  lattice  filter  methods,  can  be  generalized  to  the 
multichannel  case  by  replacing  the  various  signals  with 
signal  vectors  and  replacing  the  weighting  coefficients 
with  appropriately  dimensioned  matrices  of  coefficients. 


48 


A  discussion  of  this  appears  in  Robinson  [Ref.  45].  The, 
equations  that  describe  the  AR  and  MA  models  and  their  MMSE 
solutions  are  repeated  here  for  convenience. 


y(k)  =  2  b(i)  y(k-i) 
i=l 


(2.40a) 


.  (0) 

R  (-1)  .  . 

•  .  .  R 

yy 

yy 

yy 

.  (i) 

R  (0)  .  . 

.  .  .  R 
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R  (1) 
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R  (2) 
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R  (N) 
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(2.40b) 


Ruu(0) 


Ruu(1) 


y(k)  =  £  a(i)  u(k-i) 
i  =  0 


. Ruu(-N>  a(0) 


R  (0) 
uu 


Ruu(1-N) 


a(l) 


(2.41a) 


Ruy<0> 


Ruya) 


RuuCN)  Ruu(N-1)  '  •  •  •  Ruu<0)  a(N)  RuyCN) 


(2.41b) 


[  1 


In  a  multichannel  generalization,  y(k)  becomes  a  vector  of 
Q0  output  signals,  u(k)  becomes  a  vector  of  input  signals, 
b(i)  becomes  a  square  matrix  of  Qq  x  Qq  coefficients  and  a(i) 
becomes  a  x  Q,  matrix  of  coefficients  so  that  the  N-th 
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order  multichannel  model  equations  can  be  written  as 


AR 


N 


£(k)  =  2  b(n)  ^(k«n) 
n=l 


(2.42a) 


MA 


£<k)  =  £  a(n)  u(k-n! 


(2.42b) 


n=0 


The  equations  for  the  MMSE  solutions  (2.40b)  and  (2.41b) 
generalize  directly  as  well  by  replacing  each  correlation 
coefficient  R  (n)  by  matrices  of  correlation  coefficients 

V  W 

given  by 


R  (n)  *  e{v(k)  w(k+n)T>  (2,43) 

V  W  "  "■ 

where  v(k)  and  w(k)  are  signal  vectors.  This  causes  the 
overall  correlation  matrices  to  take  on  a  block  Toeplitz 
structure.  The  transfer  function  relationships  of  the  AR 
prediction  error  model  and  the  MA  model  take  the  form  of 
matrix  polynomials 

B(z)  =  b(l)  z"1  +  •••  +  b(N)  z"N  (2.44a) 

A(z)  =  a( 0)  +  a( 1)  z"'1  +  •••  +  a(N)  z"N  (2.44b) 

so  that 

E(z)  =  Cl  -  B ( z) ]  Y(z)  (2.44c) 
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( 2 . 44d) 


Y(z)  =  A(z)  U(z) 


where  E(z)  is  the  transform  of  the  multichannel  AR  prediction 
error  vector  e(k)  =  ^(k)  -  £(k) ,  Alternately,  equations 
(2.44a)  and  (2.44b)  can  be  written  as  single  matrices,  whose 
entries  are  polynomials  in  z  rather  than  scalars.  B(z)  is 
of  necessity  a  Qq  x  Qq  square  matrix  polynomial  while  the 
dimensions  of  A(z)  (Qq  x  (J.)  depend  upon  the  number  of 
inputs  and  outputs  which  need  not  be  the  same. 

In  the  single  channel  AR  problem,  B(z)  provides  the 
transfer  function  of  the  prediction  error,  or  inverse  fil¬ 
ter,  and  must  be  inverted  to  obtain  the  all  pole  synthesis 
filter.  The  stability  of  the  synthesis  model  therefore 
depends  upon  the  roots  of  this  polynomial.  The  matrix 
polynomial  [I-B(z)]  in  the  multichannel  AR  problem  is,  in 
like  fashion,  an  inverse  filter  representation  and  must  be 
inverted  to  obtain  the  synthesis  model.  This  inversion  of 
a  matrix  with  polynomial  entries  is  defined  in  the  same 
manner  as  the  inversion  of  a  square  matrix  with  scaler 
entries,  To  see  what  this  inverse  matrix  polynomial  looks 
like  consider  as  an  example  a  two  channel  autoregression 
with  a  prediction  error  filter  given  by 


Cl-B(z)]  = 


'21 


bll(z) 

-b12Cz) 

Cz) 

1  -  b22 

Fwpw 


ii»»«  UI>-UW>W^ 


Applying  Cramer's  rule,  the  inverse  matrix  polynomial  is 
written  as 

[I-B( z) l”1  = 

and  it  is  apparent  that  the  stability  of  the  multichannel 
synthesis  model  is  dependent  upon  the  locations  of  the  zeros 
of  the  polynomial  detCl-B(z)], 

This  straightforward  generalization  of  the  AR  and  MA 
models  is  what  has  customarily  been  used  in  the  literature 
to  develop  the  multichannel  models  and  similarly  derived 
generalizations  of  the  Levinson  algorithm  to  solve  them 
recursively  in  order  are  available  as  well.  [Refs.  57  and  61] 
The  multichannel  AR  and  MA  modeling  problems  and  their 
solutions  via  the  Levinson  algorithm  can  however  be  recast 
as  shown  in  Appendix  A  to  make  them  compatible  with  the  form 
of  other  linear  and  nonlinear  modeling  problems .  To  avoid 
confusion  later  in  the  application  of  the  results,  the 
derivations  in  Appendix  A  have  been  carried  out  in  a  generic 
form  with  x  and  d  used  to  represent  some  of  the  signals  and 
coefficients  respectively.  The  symbols  u,  y,  a  and  b  have 
been  reserved  to  denote  system  input,  output  and  weighting 
coefficients . 

Equations  (A. 7)  and  (A. 26)  provide  the  MMSE  solutions  to 
the  multichannel  AR  and  MA  modeling  problems  in  forms 
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different  than  (although  entirely  equivalent  to)  those 
resulting  from  the  straightforward  generalizations  of 
equations  (2,40b)  and  (2,41b),  The  multichannel  generali¬ 
zation  of  the  Levinson  algorithm  derived  in  Appendix  A 
can,  with  one  exception,  be  seen  as  a  matrix  algebra 
generalization  of  the  single  channel  algorithm  and  as,  one 
would  suspect,  the  single  channel  algorithm  results  as  a 
special  case  of  Appendix  A,  The  one  exception  is  that  in 
the  multichannel  case,  the  n-th  order  forward  and  backward 
prediction  error  models  are  not  simply  related  to  one 
another.  The  single  channel  AR  backward  predictor  is  given 
by  z-nB^n^ (z~^)  but  in  the  multichannel  case  the  backward 
prediction  is  not  z"n[I- £)^n^ ( z”1) ] . 

Because  of  this,  two  reflection  coefficient  matrices 
K  and  R  are  required  at  each  stage  in  the  recursion  to 
relate  the  n-th  and  (n+l)-3t  order  solutions  rather  than 
just  one  as  in  the  single  channel  problem.  Also,  in  the 
single  channel  case,  the  fact  that  jB^n^(z)|  =  |B^n^(z)| 
and  therefore  s {e^ (k) ^ }  =  e  {e^n^ (k) ^ }  made  possible  the  de¬ 
rivation  of  the  Burg  algorithm  and  the  Itakaura-Saito  al¬ 
gorithm,  which  ensured  the  magnitude  of  the  reflection 
coefficient  was  bounded  by  unity  and  that  equation  (2.19) 
would  result  in  nonnegative  values  of  MSE.  In  the  multi¬ 
channel  algorithm  however,  the  forward  and  backward  predic¬ 
tion  error  covariance  matrices  and  their  traces  are  not  the 
same  (except  for  and  and  as  a  result,  straightfor¬ 

ward  generalizations  of  the  Burg  and  Itakaura-Saito  algorithms 
are  not  possible. 
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Consequently  with  correlation  estimates  obtained  by  averaging 
over  finite  time  intervals,  there  are  no  guarantees  that 
equations  (A. 18a)  and  (A. 13b)  maintain  the  positive  definite¬ 
ness  of  the  prediction  error  covariance  matrices. 

Multichannel  generalizations  of  Burg’s  algorithm  due  to 
Nuttal  [Refs.  40  and  41],  Morf  [Ref.  39],  and  Strand 
[Ref.  50]  which  guarantee  the  positivity  of  the  covariance 
matrices  ,  are  available  but  are  not  explored  here  because  * 
of  their  complexity  and  because  they  would  take  the  dis¬ 
cussion  too  far  afield. 

The  relations  of  equations  CA.20)  and  (A. 30)  which  are 
repeated  here  for  convenience  permit  the  construction  of 
the  multichannel  AR  analysis  and  synthesis  lattice 
structures  and  the  MA  lattice  structure.  For  the  multi¬ 
channel  AR  model, 

eCn+1)(k)  =  e(n)(k)  -  K(n+1 )Te(n) (k-1)  (2.45a) 

e(n+T)(k)  =  e(n)(k-l)  -  K(n+1)  e(n)(k)  (2.45b) 

e(0)(k)  =  e(0)(k)  =  x(k)  (2.45c) 

and  the  corresponding  prediction  error  lattice  is  shown  in 
Figure  2,8. 


(k) 


(k) 

Figure  2.8.  Multichannel  AR  prediction  error  lattice 
structure  for  a  second  order  model.  All 
signal  paths  are  vector  paths  and  summa¬ 
tions  are  vector  summations.  The  multi¬ 
plications  indicate  px-emultiplication  of 
the  signal  vector  by  the  specified  coef¬ 
ficient  matrix. 

To  obatin  the  multichannel  AR  synthesis  lattice,  equation 
(2.45a)  can  simply  be  rewritten  as 

e(n)(k)  =  eCn+1)(k)  +  K(n+1)Te(n) (k-1)  (2.46) 

resulting  in  the  structure  shown  in  Figure  2.9. 


Figure  2.9.  Multichannel  AR  synthesis  lattice 

structure  for  a  second  order  model. 
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For  the  multichannel  MA  model,  equations  (2.45)  and 
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As  a  consequence  of  the  successive  decoupling,  the  forward 
and  backward  reflection  coefficient  matrices  at  the  (n+l)-st 
stage  can  be  set  to  minimize  the  trace  of  the  forward  and 
backward  prediction  error  covariance  matrices  respectively, 
given  the  best  lattice  of  order  n.  This  provides  alterna¬ 
tives  to  equations  (A, 17)  for  calculating  K  and  K  and  also 
generalizes  the  forward  and  backward  single  channel  solutions 
discussed  previously,  resulting  in 

K(n+1)  =  ?<n)-1  A<n)T 

pr(n+l)  _  p(n)"1  A<n) 

where 

A(n)  =  e{e<n)(k)  eCn)(k-l)T)  (2.49c) 

It  is  also  possible  to  show  that  these  relationships  are 
entirely  equivalent  to  equation  (A. 17).  In  the  multi¬ 
channel  MA  lattice ,  the  orthogonality  of  the  backward  pre¬ 
diction  error  signals  also  allows  the  G  matrices  to  be 

calculated  in  succession  providing  an  alternative  to  equation 

(A. 28b)  and  generalizing  the  single  channel  solution  given 
by  equation  (2.39).  Setting  to  minimize  the  trace  of 

the  error  covariance  matrix 

PQ(n)  =  e{eQ(n)(k)  eg(n)(k)?}  (2.50a) 


(2.49a) 

(2.49b; 
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where 


eQ(n)(k)  =  £(k)  «  £(n)(k)  (2.50b) 

results  in  a  solution  given  by 

‘  3(n)  =  |(n)  1  e{e(n)(k)  £(k)T}  (2.50c) 

Another  important  characteristic  of  the  lattice  solutions 

t 

to  the  AB.  and  MA  modeling  problems  given  by  equations  (2.49) 

I 

and  (2.50)  and  their  single  channel  counterparts  is  that  they 
do  not  impose  any  requirements  to  window  the  data  when  finite 
time  averages  are  used  to  estimate  correlations.  The  auto¬ 
correlation  function  of  a  signal  is  inherently  an  even  func¬ 
tion  so  that  R  (n)  =  R.„,(~n),  This  fact  is  responsible  for 
much  of  the  special  structure  of  the  correlation  matrices 
that  appear  in  the  model  solution  equations,  and  was  also 
used  in  the  derivation  of  the  Levinson  algorithm,  In  esti¬ 
mating  the  autocorrelation  function  via  time  averaging  over 
a  finite  interval,  a  window  function  that  is  nonzero  over 
only  a  given  interval  must  be  applied  to  the  data  to  retain 
this  even  symmetry  property  in  the  estimate.  If  the  data  is 
not  set  to  zero  outside  a  given  interval,  end  effects  will 

A  A 

destroy  the  symmetry  so  that  Rvv(n)  i  Rvv(-n)  as  depicted 
in  Figure  2.11. 
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I  Averaging  Interval 


v(t> 


v(t-n) 


v( t+n) 


Figure  2.11.  Time  averaging  to  estimate  correlations 
without  windowing. 

In  the  lattice  solutions  of  equation  (2.49)  however,  there 
is  no  requirement  to  make  such  an  artificial  assumption 
about  the  data  (that  it  is  zero  outside  some  interval). 

These  properties  of  the  lattice  solution  methods  were 
responsible  for  their  initial  use  by  Burg  in  his  work  on 
maximum  entropy  spectral  analysis  [Ref.  5]  and  have  con¬ 
tinued  to  generate  interest  in  the  application  of  lattice 
methods  to  other  types  of  modeling  problems. 
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F.  ADAPTIVE  MODELING 


The  LMS  adaptive  algorithm  provides  a  well  known  alter¬ 
native  method  for  obtaining  the  solution  to  the  AR  or  MA 
modeling  problems  which  does  not  require  the  estimation  of 
correlations  or  the  inversion  of  a  matrix  [Refs.  58,  59 
and  60].  This  algorithm  updates  an  estimate  of  the  model 
solution  vector  at  each  time  instant  by  an  amount  propor¬ 
tional  to  the  negative  of  the  instantaneous  gradient  of  the 
cost  function;  i.e. ,  in  a  MA  model, 

a+ (k+1)  s  a+(k)  -  y  V+(k>  (2.51a) 

where  u  is  a  proportionality  constant  or  adaptive  gain. 

Since  the  actual  gradient  is  usually  not  known,  it  is 
approximated  by  using  the  square  of  a  single  sample  of  the 
error  as  an  estimate  of  the  MSE  so  that 

»\k>  - 

3a 

and 


=  -2  u+(k)  e(k)  (2.51b) 

a  =a  (k) 


a+(k+l)  =  a+(k)+2y  u+(k)  e(k)  (2,51c) 

In  each  of  the  models  considered  here,  the  cost  function 
(MSE  or  trace  P)  is  a  quadratic  function  of  the  model  weights 
and  defines  a  concave  hyperparaboled  with  a  unique  minimum. 
The  functioning  of  the  LMS  algorithm  under  these  conditions 
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can  easily  be  understood  by  considering  the  scalar  case  of 
equation  (2,51)  illustrated  in  Figure  2,12. 

As  this  illustration  shows  ,  the  algorithm  can  actually 
diverge  for  too  large  a  value  of  adaptive  gain.  The  rate 
of  convergence  is  also  dependent  on  the  size  of  the  adaptive 
gain.  Widrow  has  shown  that  for  stability,  the  gain  must  be 
set  so  that 

0  <  u  <  (2.52a) 

Amax 

While,  in  the  mean,  the  weight  vector  converges  with  an 
exponential  time  constant  of 

X  5  <2'52b) 

where  and  ^max  are  the  smallest  and  largest  eigenvalues 

of  the  input  autocorrelation  matrix  R  +  +.  From  the  stand- 

"u  u 

point  of  stability,  y  should  be  made  relatively  small  but 
for  rapid  convergence,  equation  (2.52b)  dictates  that  it 
should  be  large.  Setting 


U 


a 

^max 


(2.53a) 


where  a  is  a  normalized  gain  and  0  <  a  <1,  equation  (2.52b) 
becomes 


y 


Snax 

Xmin 


(2.53b) 


EK-'i  i 
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b)  Intermediate  adaptive  gain*,  oscilatary  convergence  toward 


c)  Large  adaptive  gain;  divergence  away  from  the  solution 


Figure  2.  . 2.  Behavior  of  the  LMS  adaptive  algorithm 
for  various  adaptive  gains. 


and  for  a  wide  disparity  between  the  largest  and  smallest 
eigenvalues  (Xm^n  <<•  convergence  will  be  quite  slow. 

Tuis  consideration  becomes  increasingly  important  when  high 
order  model  solutions  are  obtained  adaptively  since  the 
dimensonality  of  the  input  autocorrelation  matrix  will  be 
high  and  the  possibility  of  a  wide  eigenvalue  disparity 
greater. 

These  same  adaptive  techniques  have  been  applied  by 
Griffiths  [Refs.  16,  17,  18  and  31],  to  the  AR  and  MA 
lattice  filter  structures  derived  in  the  last  section.  The 
key  difference  between  the  conventional  adaptive  filter  and 
the  adaptive  lattice  is  that  in  the  lattice,  the  adaptation 
is  carried  out  on  a  stage  by  stage  basis  for  each  of  the 
reflection  coefficient  matrices,  while  in  the  more  conven¬ 
tional  approach,  the  entire  weight  vector  is  adapted.  It 
has  already  been  established  that  the  lattice  structure 
makes  the  model  solutions  recursive  in  order.  Implementing 
the  lattice  adaptivity  makes  the  solution  recursive  in  time 
as  well  since  the  estimate  of  the  solution  at  each  instant 
is  dependent  upon  prior  estimates  of  the  solution, 

The  conventional  adaptive  filter  algorithm  forms  an 
error  signal  as  the  difference  between  some  desired  signal 
and  its  estimate*,  i.e, 

e(k)  =  y(k)  -  a*(k)Tu+(k)  (2.54) 
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where  y(k)  is  the  desired  signal,  a+  is  the  weight  vector 
and  u+(k)  is  the  input  vector,  and  the  time  update  for  the 
weight  vector  is  given  by  equation  (2,51c).  To  derive  the 
adaptive  AR  lattice  consider  equations  (2.45)  for  a  single 
stage.  The  lattice  in  general  has  vecto  error  and  desired 
signals  and  coefficient  matrices  as  opposed  to  scalar  error 
and  desired  signals  and  a  coefficient  vector  in  equation 
(2.54)  but  such  a  generalization  is  straightforward.  Com¬ 
paring  equation  (2.45a)  to  (2.54)  it  is  clear  that: 

1)  e^n+1\k)  is  analogous  to  the  error  signal; 

2)  e^n^(k)  is  analogous  to  the  desired  signal; 

3)  e  (k-1)  is  analogous  to  the  input  signal  vector. 
Using  the  trace  of  P^n+^  as  a  cost  function  and  applying 

a  LMS  adaptive  algorithm  to  determine  the  forward  reflection 
coefficient  matrix  it  follows  that 

K(n+1)(k+l)  =  K(n+1)(k)  +  2y(n+1)e(n) (k-l)e(n+1) (k)T 

(2.55a) 

With  these  analogies,  equations  (2.51c)  and  (2.55a)  are  seen 
to  be  virtually  identical  with  the  exception  that  (2,51c) 
uses  a  scalar  error  to  adapt  a  weight  vector  and  (2.55a)  uses 
a  vector  error  signal  to  adapt  a  coefficient  matrix. 

Proceeding  in  a  similar  fashion  with  equation  2.45b  it  is 
clear  that: 

1)  e^n+^(k)  is  analogous  to  the  error  signal; 

2)  e^(k-l)  is  analogous  to  the  desired  signal; 
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3)  e/n^(k)  is  analogous  to  the  input  signal  vector. 
With  the  trace  of  P^n+^  as  a  cost  function,  the  time 
update  relation  for  the  backward  reflection  coefficient 
matrix  is 


£(n+1)(k+l)  =  £(n+1)(k)  +  2y(n+1)  e(n)(k)  e''n+1)(k)T 

(2.55b) 

For  a  MA  lattice,  equation  (2,47)  must  also  be  considered. 
Multiplying  both  sides  of  (2.47)  by  minus  one  and  adding 
^(k)  results  in 


eQ(n+1)(k)  *  eQ(n)(k)  -  G(n+1)  e(n+1)(k) 


(2.56) 


where  eQ^n+1^(k)  is  defined  as  in  equation  (2.50) 


It  is 


evident  that! 

(n+1) 


£0 


(k)  is  analogous  to  the  error  signal; 


(n) 


2)  £q  (k)  is  analogous  to  the  desired  signal; 

3)  e^n+1^(k)  is  analogous  to  the  input  signal  vector. 

With  the  trace  of  Pg^*1^  as  a  cost  function,  the  time  up- 

(  1  } 

date  relation  for  G  is  given  by 


S(nn)!M)  =  GCn+1)Ck>  +  2g'ntl)  I(n+1)(k)  en(n+1)(k>T 

—  —  g  —  — 0 

(2.57) 
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It  is  significant  to  note  that  three  different  adaptive 
gains  have  been  used  in  equations  (2,55)  and  (2.57)  and 
that  the  gains  have  been  superscripted  indicating  that  they 
vary  from  one  lattice  stage  to  the  next.  For  stability  consi¬ 
derations  the  adaptive  gain  used  in  the  LMS  algorithm  must 
satisfy  equation  (2.52a)  and  therefore  is  related  to  the 
largest  eigenvalue  of  the  input  autocorrelation  matrix  by 
equation  (2.53a).  In  developing  the  time  update  relations 
for  the  lattice  coefficients,  three  different  input  signals 
were  used  and  these  inputs  also  differ  from  one  lattice 
stage  to  the  next.  Indeed,  even  for  the  case  where  the  in¬ 
put  x(k)  to  the  lattice  structure  is  stationary,  the  inputs 
to  all  lattice  stages  except  the  first  are  nonstationary 
since  these  inputs  are  outputs  of  other  lattice  stages.  This 
fact  indicates  that  time  varying  adaptive  gains  are  appro¬ 
priate  as  well  in  equations  (2.55)  and  (2.57),  Equation 
(2,53a)  is  of  no  direct  usefulness  however  in  setting  the 
adaptive  gains  since  the  time  varying  eigenvalues  are  not 
known.  Recognizing  that  the  largest  eigenvalue  is  always 
less  than  the  trace  of  the  input  autocorrelation  matrix 
(which  is  a  measure  of  the  power  in  the  input  signal  vector) 
the  gains  can  be  set  as 

y(n+1)(k)  s  — _a -  (2.58a) 

Wk) 
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(2.58b) 


-(n+l) 


(k) 


(n+l) 


(k) 


(2.58c) 


2—2  2 

where  an+]_(k)  ,  an+l^  and  Yn+1^^  are  es't^ma'tes  the 
power  in  the  three  input  signal  vectors  and  a  is  a  normalized 
adaptive  step  size  with  0  <  a  <  1,  A  method  that  has  commonly 
been  applied  to  obtain  these  estimates  is  to  employ  a  first 
order  low  pass  filter  so  that 


Wk+1>2 

=  Cl-alan+1(k)2+ae(n)(k-l)Te(n,)(k-l) 

(2.59a) 

Vi(kn,! 

=  Cl-a]0n+1(k)2+ae(n>(k)Te(n)(k) 

(2.59b) 

wk+1)2 

=  Cl-a]yn+1(k)2+ae0(n)(k)Te0(n)(k) 

(2.59c) 

Taken  together,  equations  (2.55),  (2.57),  (2.58)  and  (2.59) 
define  the  adaptive  solutions  for  the  AR  and  MA  multichannel 
lattice  models . 

To  understand  the  potential  advantage  offered  by  the 
adaptive  lattice  form,  recognize  that  while  the  conventional 
approach  solves  a  high  order  minimization  problem  by  adapting 
all  the  coefficients  at  once,  the  lattice  breaks  the  problem 
down  into  a  succession  of  lower  order  minimizations  at  each 
stage  and  solves  these  lower  order  problems  adaptively.  The 
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dimensionality  of  the  input  autocorrelation  matrices.,  at 
each  lattice  stage  in  general  is  significantly  less  than 
that  of  the  large  input  autocorrelation  matrix  in  the  con¬ 
ventional  adaptive  algorithm  and  consequently  it  is  hoped 
that  the  possibility  of  a  large  eigenvalue  disparity  with 
its  attendant  slow  convergence  is  reduced. 

This  advantage  is  most  evident  in  a  single  channel 
adaptive  lattice  where  the  inputs  at  each  stage  are  single 
signals  and  »ir  corresponding  autocorrelation  matrices 
are  lxl  in  dimension.  In  this  case  the  ratio  of  smallest 
to  largest  eigenvalues  is  unity  and  the  convergence  of  each 
stage  is  quite  rapid  while  the  convergence  of  the  overall 
model  is  independent  of  the  eigenvalue  ratio  for  the  over¬ 
all  higher  dimension  input  autocorrelation  matrix.  This  has 
been  demonstrated  by  Satorius  [Refs.  46  and  47]  who  has 
shown  that  the  single  channel  adaptive  lattice  converges 
much  more  rapidly  than  the  corresponding  conventional  adap¬ 
tive  filter,  and  does  so  independently  of  the  eigenvalue 
ratio  on  the  overall  input  channel  autocorrelation  matrix. 

Furthermore  in  a  single  channel  adaptive  lattice,  the 
time  update  relations  are  simplified  by  the  fact  that  the 
forward  and  backward  reflection  coefficients  are  the  same. 
Using  the  average  of  the  mean  square  values  of  backward 
and  forward  prediction  errors  as  a  cost  function  and  applying 
an  adaptive  algorithm  it  follows  that 
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+e(n)(k-l)e(n+1)(k)]  (2.60a) 

where 

an+i(k+l)2=C1-a]an+iO<)2+|[e(n)(k)2+e<n)(k-1)2]  (2.60b) 

The  very  nature  of  the  lattice  structure  however  with 
the  output  of  one  stage  providing  an  input  to  the  next  stage, 
greatly  complicates  the  analysis  of  the  convergence  proper¬ 
ties  of  the  adaptive  lattice  model.  Ever,  when  x(k) ,  the 
input  to  the  lattice,  is  stationary,  inputs  to  all  stages 
except  the  first  are  nonstationary.  An  approximate  analysis 
of  convergence  and  stability  on  a  stage  by  stage  basis  is 
possible  if  it  is  assumed  that  all  prior  stages  have  con¬ 
verged  and  are  providing  stationary  inputs  to  the  stage 
under  investigation.  With  this  assumption,  the  adaptive 
solution  for  the  K^n+^  ,  j<^n+^  and  G^n+^  matrices  are 
obtained  from  the  operation  of  three  independent  LMS  algo- 
rithms  as  shown  earlier,  with  inputs  given  by  ev  '(k-1), 
e^n^(k)  and  e^n+^^ (k),  respectively .  Stability  limits  on 
the  adaptive  gains  used  in  the  stage  and  the  convergence 
properties  of  the  stage  are  then  determined  by  the  eigen¬ 
values  of  the  and  matrices.  A  more  exact 

analysis  of  the  properties  of  the  adaptive  lattice  that 
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III.  ARMA  MODELING 


One  of  the  most  serious  disadvantages  of  either  AR  or 
MA  modeling  is  the  fact  that  to  adequately  represent  even 
simple  linear  systems,  both  methods  may  require  a  large 
number  of  parameters  (a  high  order  model) .  This  problem 
arises  since,  from  a  transfer  function  standpoint,  AR  and 
MA  models  attempt  to  model  the  system  using  only  poles  or 
only  zeros,  in  spite  of  the  fact  that  the  physical  system 
may  have  both  zeros  and  poles.  While  modeling  the  effects 
of  a  zero  with  a  number  of  poles  and  visa  versa  can  be 
analytically  justified  as  shown  in  the  previous  chapter, 
it  makes  far  more  sense  (both  from  the  viewpoint  of  model 
accuracy  and  efficient  use  of  model  parameters)  to  let  the 
model  represent  the  system  as  it  really  is  with  both  zeros 
and  poles  if  this  is  at  all  possible.  The  ARMA  model  is  a 
generalization  of  the  AR  and  MA  models  and  accomplishes 
exactly  this,  representing  the  system  in  rational  transfer 
function  form. 

It  is  worth  noting  that  the  titles  of  all  pole  and  all 
zero  modeling  that  have  been  associated  with  AR  and  MA 
modeling  are  misnomers.  Both  have  equal  numbers  of 
zeros  and  poles.  In  the  AR  model  however,  all  the  zeros 
occur  at  the  origin  of  the  z-*plane  as  do  the  poles  of  a.  MA 
model.  The  ARMA  model  removes  these  constraints. 
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After  a  brief  discussion  of  two  alternate  ARMA  modeling 
methods  due  to  Shanks  and  Prony,  the  equation  error  formu¬ 
lation  for  ARMA  modeling  is  developed  and  the  new  results 
presented.  Model  transition  formulas  relating  the  ARMA 
model  to  the  MA  and  AR  models  are  developed  and  the  input 
signal  requirements  of  the  modeling  process  explored.  It 
is  shown  that  after  suitable  modification,  the  Levinson 
algorithm  can  be  applied  to  solve  the  ARMA  modeling  problem 
recursively  in  order  and  lattice  solution  methods  are  also 
developed  for  both  a  batch  processing  and  an  adaptive  model 
solution,  The  results  of  experimental  simulations  of  both 
of  these  modeling  solution  methods  are  presented  and  dis¬ 
cussed,  and  comparisons  are  made  with  conventional  means  of 
ARMA  modeling  using  the  equation  error  formulation.  Finally 
it  is  also  shown  that  the  lattice  solution  methods  can  be 
generalized  to  solve  for  the  multichannel  ARMA  model  with 
arbitrary  numbers  of  inputs  and  outputs. 


A.  LINEAR  ARMA  MODELING  AND  ITS  RELATION  TO  AR  AND  MA 
MODELING 

The  ARMA  model  for  linear  systems  assumes  the  current 
value  of  the  output  of  the  system  is  given  by  a  weighted 
combination  of  present  and  past  values  of  the  input  and 
past  values  of  the  output.  In  terms  of  the  discussions  of 
Chapter  I,  F^q  is  assumed  to  be  zero  and  F1Q  and  F2Q  take 
on  the  following  forms 


F10Cu(k)] 


£  <*(n)  u(k-n) 

n=0 


(3.1a) 
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F2Q[y(k-l)]  = 


(3.1b) 


V 

2  b(n)  y(k-n) 
n=l 

leading  to  a  transfer  function  representation  for  the  system 
given  by 

M  a(n) z"n 

=  n°,°-  v - - -  =  (3.1=) 

1-  fa(n)z 
n=l 

A  number  of  methods  exist  for  finding  the  model  coef¬ 
ficients  (a(n)  }  and  (b(n)}.  As  stated  in  Chapter  I,  a  MMSE 
solution  via  the  direct  form  modeling  approach  requires  the 
solution  of  a  system  of  highly  nonlinear  equations  and  in 
general  is  untractable.  An  alternative  is  to  first  obtain 
an  estimate  of  the  denominator  polynomial  8(z)  by  some 
means  such  as  AR  modeling  and  then  using  this  in  the  system 
shown  in  Figure  3,1,  estimate  the  numerator  polynomial  A(z) 
by  setting  its  coefficients  to  minimize  the  mean  square 
value  of  the  error.  This  method  was  first  explored  by 
Shanks.  [Ref.  49] 

Another  alternative  is  to  apply  the  Prony  method  [Refs. 

8,  52  and  56]  derived  in  Appendix  E  which  obtains  the  model 
parameters  by  matching  the  impulse  response  of  the  system 
and  model  over  the  first  W+M+l  sample  intervals,  Both  of 
these  techniques  share  a  common  characteristic.  They  both 
start  by  independently  estimating  the  denominator  coefficients 
(or  model  poles)  and  then,  given  this  estimate,  solve  for 


73 


u(k) 


/  / 

* - / 


e(k) 


Figure  3.1.  Shanks  method  for  ARMA  modeling 

•the  numerator  coefficients  (or  model  zeros).  This  is  in¬ 
tuitively  unappealing  in  that  one  would  expect  these  two 
estimation  problems  to  be  more  closely  coupled  with  the 
zero  estimates  also  affecting  the  estimates  of  the  poles. 

The  application  of  the  equation  error  formulation  to  the 
ARMA  modeling  problem  permits  simultaneous  estimation  of  the 
model  zeros  and  poles  and  was  first  used  by  Kalman  [Ref.  23] 
in  work  on  self -optimizing  control  systems.  The  prediction 
error  form  of  the  model  is  considered  here  where  is  set 
to  zero  while 

M 

Fin[u(k)]  =  V  a(n)  u(k-n)  (3.2a) 

1U  n=0 

M 

F2Cy(k)l  =  y(k)-  £  b(n)  y(k-n) 


(3.2b) 


[■he  analysis  model  is  depicted  in  Figure  3.2  where 


^M'TiirfT'^Tr" -  " 


The  expression  for  the  model  error  can  be  written  as 


eQ(k)  =  y(k)  -  C^(k)T  j  ,y+(k)T] 


(3.4a) 


where  ^(k)  and  b  are  defined  as  in  equation  (2.7)  and 


u+(k)  =  Cy(k)  «•«  u(k-M)]^ 


(3.4b) 


a+  =  Ca(0)  • • •  a(K) ]T 


(3.4c) 


This  results  in  an  expression  for  the  mean  square  error 
which  is  a  quadratic  function  of  the  model  coefficients, 
with  the  MMSE  solution  for  those  coefficients  given  by 


'  t 


£yy  '  -  + 

i  yu 


£  +  1  *  +  + 
u  y  ,  u  u 


E2  .  *  Rvy<0) 

min 


T  I 

[bT  t  a  ] 


(3.5a) 


(3.5b) 


1.  Model  Transition  Relationships 


Comparing  equations  (3.4)  and  (3.5)  with  their  AR  and 


MA  counterparts,  it  is  clear  that  the  ARMA  model  provides  a 
generalization  of  these  other  models  since  they  can  be 
obtained  from  the  ARMA  model  by  assuming  that  either  a+  or  b 
is  zero.  Consequently  it  is  susceptible  to  the  same  type  of 


bias  introduced  in  the  AR  and  MA  models  by  the  presence  of 
additive  noise  on  either  the  system  input  or  output  signals. 
To  develop  the  relationships  between  these  models  further, 
consider  the  inversion  of  the  correlation  matrix  in  equation 
(3.5a)  in  terms  of  its  component  matricies.  Since  the  left 
and  right  inverses  of  a  nonsingular  square  matrix  are  the 
same,  either 


can  be  used  to  find  the  required  inverse  in  partitioned  form. 
Solving  for  the  right  inverse  of  equation  (3.6b)  yields 


(£yy  -  £  +£'i  ♦  £  ♦  )_1 

yu  u  u  u  y 

(3.7a) 

•£yi  £  ♦<£  +  +  -£♦  £  S'1 

yu  u  u  u  y  yu 

(3.7b) 

-£‘i  +  £  *  <£yy  -  £  ♦  £"i  +  £  *  >_1 

uu  uy  **  yu  uu  uy 

(3,7c) 

^5.  +  +  "  5,  +  5yy  5. 
u  u  u  y  yu 

(3 , 7d) 
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Solving  for  the  left  inverse  of  equations  <3. 6a)  gives  iden¬ 
tical  results  for  A  and  D  while  equivalent  but  different 
forms  are  obtained  for  B  and  C  given  by 


S  ’.-(Ryy-S  +*■}  +£  +  J'1  S  Xl  * 

yy  yu  u  u  u  y  yu  u  u 


(3.8a) 


£  *  -<£  ♦  .-5  ♦  ?  R  .J'1  5  +  S,'1 

uu  u  y  ”  yu  uy 


yy 


(3.8b) 


Using  equations  (3.7)  in  equation  (3.5a),  the  solutions  for 
the  ARHA  coefficient  vectors  are  given  by 


fe  *  (Ryv-RyU^'J  +S  ♦  >  ^  £ 

yy  y  u  u  u  y 


•yy 


5  +<5  +  +*R  ♦  +>_1  £  + 

yu  uu  u  y  yu  uy 


(3.9a) 


and 


a*  =  “R”+  +  £  +  <£yy-EyU*R  +  +  E  +  J'^yy 
uu  u  y  *  uu  uy  yy 


+  (R 


-  R  .  R"1  R  J.)"1  r 


-  +  +  -  +  -yy  -  + '  £.  + 

uu  u  y  yu  uy 


(3.9b) 


The  matrix  inversion  lemma  [Refs.  11  and  19]  which  states 
that 


(E+FGH)"1  =  E"1  -  E~1F(G~1+HE~1Fr1  HE"1 


(3,10) 


for  nonsingular  square  matrices  E,  G  and  E+FGH,  can  be 
used  in  equation  (3.9)  to  rewrite  them  as 


k  *  €y  Eyy  *  ^y  £  ♦  <£  +  +  '  5  +  £yy  £  ♦  >' ^ 

J  J J  J  J  yu  U  U  u  y  J J  yu 


CS.  ♦  £yy  £yy  '  S  +  3 
u  y  u  y 


(3.11a) 
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H  +  +  £  +  “  Eyy^ 
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(3.11b) 


The  all  zero  and  all  pole  model  solutions  of  corresponding 
orders  however  are 


-  AZ  5  -uV  Vy  and  ***  ' 


where  subscripts  are  used  to  distinguish  these  solutions 
from  their  counterparts  in  the  ARMA  zero  pole  model. 

From  equations  (3,11)  it  follows  that 


£zp  5  Sap  *  £yy  R  ♦'£..-£.  Ky  E  ^ 
^  yu  uu  uy-^yu 


C£  +  £ap  -  E  ♦  ] 

u  y  u  y 


(3,12a) 
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yu 


~AZ  "  £yy] 


(3.12b) 


Following  a  similar  development,  the  left  inverse  relation¬ 
ships  of  equations ( 3 , 8 )  can  be  used  to  write 


—  +  5.  +  +  5.  +  ^ 

yu  u  u  u  y 


+-AZ] 


(3.12c) 


+  + 
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R  t  ^5  +)'1  l>  + 

u  y  jjr  yu  u  y 


R  .  bAp]  ( 3 . 12d ) 
u  y  _A^ 


Equations  (3.12)  are  termed  the  "Zero  Pole  Model  Transi¬ 
tion  Formulas"  and  specify  the  relationships  between  the 
various  models.  It  is  interesting  to  note  that  equations 
(3.12a)  and  (3.12b)  take  the  form  of  a  linear  observer  with 
a  new  estimate  of  the  solution  given  by  the  old  estimate 
plus  a  gain  times  an  error  term.  To  gain  some  insight  into 
the  functioning  of  these  formulas,  consider  the  form  of  R  (n) 
and  Ruy(n)  for  the  linear  system  described  by  the  transfer 
function  of  equation  (3.1c) 

N  M 

Ryy(-n)  =  £  b(i)  Ryy(-i-n)  +  £  a(i)  Ryu(-i-n) 

L=1  L=0 

(3.13a) 
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R  (-n)  =  V  b(i)  R, ,„(-i-n)  +  a(i)  R  (-i-n) 

uy  LmJ  uy  iLu  uu 

L*1  iaO 


(3.13b) 


Assuming  that  N=N  and  M=H,  and  writing  equation  (3.13a)  for 
-N<n<~l  and  equation  (3.13b)  for  -M<n_<C)  results  in 


b  +  P.  + 
~yu 


(3.14a) 
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fa  f 
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*  +  +  £ 
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(3.14b) 


These  constraints  on  the  system  input  and  output  auto  and 
cross  correlation  coefficients  are  the  ARMA  modeling  equa¬ 
tions  of  (3.5a)  with  the  model  coefficients  replaced  by  the 
system  parameters.  In  AR  modeling,  bAp  is  set  to  satisfy 
the  constraints  of  equation  (3.14a)  with  the  assumption  that 
a+  is  zero.  The  error  term  in  the  model  transition  formula 
(3.12a)  then  checks  this  solution  to  see  if  it  also  satisfies 
the  constraints  of  equation  (3,14b)  still  assuming  that  a+ 
is  zero . 


error  =  R  t  bAp 
u  y 


£  + 
u  y 


(3.15) 
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If  this  error  is  zero  and  the  constraints  of  equation  (3.14b) 
are  satisfied ,  equation  (3.12a)  sets  bzp  s  bAp  and  equation 

.f 

(3,l?d)  acts  a  to  zero.  If  however  the  error  is  nonzero, 

(3.12a)  adjusts  b^p  in  proportion  to  the  error  to  obtain 

*  + 
h_2P  and  (3.12d)  then  provides  a  nonzero  azp.  Thus  equations 

(3.12a)  and  (3.12a)  are  complementary,  specifying  the  ARMA 
or  zero  pole  model  solution  of  order  M  over  N  when  given 
the  N-th  order  AR  or  all  pole  model  solution. 

In  like  mannt*  ,  equations  (3.12b)  and  (3.12c)  give  the 
ARMA  model  solution  of  order  M  over  N  when  given  the  M-th 
order  MA  or  all  zero  model  solution.  The  all  zero  solution 
is  obtained  from  equation  (3.14b)  assuming  that  b  is  zero. 
Equation  (3.12b)  checks  this  solution  against  the  con¬ 
straints  imposed  by  equation  (3.14a)  with  the  same  assump- 

4*  4* 

tiona  and  adjusts  a.^  appropriately  to  determine  azp. 

Equation  (3.12c)  then  sets  bzp>  completing  the  zoro  pole 

model  solution . 

2 .  Modeling  Input  Signal  Requirements 

Another  aspect  of  the  modeling  problem  that  must  ba 
considered  is  that  of  system  id jntifiability .  If,  from  the 
available  measurements  of  signals,  a  model  can  be  obtained 
that  accurately  represents  the  system's  operation,  the 
system  is  considered  identifiable.  The  two  issues  that 
arise  therefore  are  the  measurement  requirements  (which 
signals  must  be  measured)  and  requirements  on  the  input 
signal  used  to  excite  the  system  during  the  modeling 
process.  In  the  equation  error  formulation  of  the  /'RMA 
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model,  both  existing  signals  (system  input  and  output)  must 
be  observed  (or  at  least  a  knowledge  of  their  auto  and  cross 
correlation  functions  must  be  available).  Most  discussions 
of  input  signal  requirements  for  identif iability  simply 
state  that  the  system  can  be  identified  if  the  input  signal 
is  sufficiently  rich,  persistent  or  exciting,  eg  [Ref.  19] 

To  explore  the  question  of  input  signal  requirements  fur¬ 
ther,  consider  the  mean  square  equation  error  cost  function 
being  minimized.  Assuming  that  the  equation  error  signal 
is  ergodic  and  has  finite  energy  its  mean  square  value  is 
obtained  via  time  averaging  as 

00 

E2  a  e{e(k)2}  a  £  «(n)2  (3.16) 

n*-» 

Applying  Panevals  relation  this  becomes 
«o  n 

E2  «  £  e(n) 2  ■  J  E(ej9)E*(ei9>  d0  (3.17) 

n« -w  -7T 

where  0«wT  and  *  indicates  the  complex  conjugate.  The 
equation  error  j  represented  in  the  transform  domain  as 

E( z)  a  [B(z)H(z)  -  A( z) 3lK z)  (3.10) 

and  using  chis  in  equation  (3.17),  the  cost  function  become* 
n 

/j  2  2 

|CB<ej0)HUj9)-A(e^>|  '•  |U(e*9>|  dt)  (3,19) 
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showing  that  the  power  spectrum  of  the  input  signal  acts  as 
a  frequency  dependent  weighting  function  on  a  transfer 
function  error  term.  Therefore  to  identify  the  system 
equally  well  at  all  frequencies,  the  input  must  have  a 
flat  spectrum  as  will  be  the  case  for  a  white  noise  input 
or  an  impulse  function  input.  Otherwise,  the  model  trans¬ 
fer  function  will  only  be  matched  to  the  system  transfer 
function  over  the  range  of  frequencies  where  the  input 
signal  has  significant  power. 

As  an  example  consider  the  equation  error  ARMA 
model  for  a  fourth  order  system  driven  by  a  single  sine 
wave  input  at  a  frequency  of  it/ 3.  According  to  equation 
(3.19),  the  model  transfer  funotlon  will  only  be  required 
to  matoh  the  system  at  this  single  frequency,  and  to  accom¬ 
plish  this,  only  a  first  order  model  is  needed,  Any  in¬ 
crease  in  model  order  above  first  order  therefore  should 
have  no  effect,  Figure  3.3a  shows  a  comparison  of  the  mag¬ 
nitude  spectrum  of  the  fourth  order  system  and  its  first 
order  ARMA  model  obtained  using  the  sinusoidal  input  and 
as  anticipated  they  matoh  at  the  frequency  tt/3  (coinoiden- 
tally  they  also  match  at  one  other  frequency  as  well), 

Figure  3.3b  shows  the  same  comparison  but  with  a  fourth 
order  ARMA  model,  It  is  clear  that  increasing  the  model 
order  failed  to  improve  its  accuracy  and  that  the  model 
accurately  represents  the  system  only  at  the  frequency  of 
the  input  signal  and,  by  coincidence,  at  one  other  frequency, 
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It  should  be  noted  that  this  type  of  analysis  to 
determine  input  signal  requirements  could  be  applied  to  the 
A R  and  MA  models  as  well  resulting  in  the  same  conclusions. 


B.  A  RECURSIVE  IN  ORDER  SOLUTION  FOR  THE  ARMA  MODEL 

Since  the  equation  error  formulation  for  the  ARMA  model 
is  a  generalization  of  similar  formulations  for  AR  and  MA 
modeling,  it  is  reasonable  to  assume  that  a  Levinson-type 
algorithm  could  be  devised  to  obtain  the  ARMA  solution 
recursively  in  order,  and  that  from  that  algorithm,  lattice 
filter  methods  applicable  to  the  ARMA  modeling  problem  could 
be  derived.  Attempts  to  develop  such  an  algorithm  directly 
for  the  ARMA  modeling  equation  (3,5a),  however,  fail  to 
provide  useful  results,  The  first  problem  that  arises  is 
in  deciding  which  model  order  to  make  recursive \  the  order 
of  the  numerator  polynomial,  the  order  of  the  denominator 
polynomial  or  both.  If  it  is  assumed  that  the  numerator 
and  denominator  are  of  equal  orders  (M»N),  the  ARMA  modeling 
equations  become  as  shown  in  equation  (3.20),  However, 
efforts  to  develop  a  Levinaon-type  algorithm  for  this  system 
of  equations,  where  the  numerator  and  denominator  polynomials 
of  the  model  are  incremented  simultaneously  to  obtain  re¬ 
cursive  in  order  solutions,  are  still  fruotratea  by  the 
presence  of  the  (N+l)-st  row  and  oolumn  in  equation  (3.20), 

This  arises  because  the  numerator  coefficient  vector  is 

■** 

a  (N+D-vector  while  the  denominator  coefficient  vector  b 

•%> 

to  a  N-vector.  if  it  is  further  assumed  that  the  coefficient 
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a(0)  is  known  in  advance  or  can  be  estimated  in  some  other 
fashion,  equation  (3.20)  for  the  solution  for  the  remaining 
2N  coefficients  becomes  as  written  in  equation  (3.21). 
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(3,21) 

The  coefficient  a(0)  essentially  has  the  role  of  a  gain  for 
the  model,  and  a  method  for  estimating  it  after  all  the  other 
coefficients  ’are  obtained. (as  was  done  in  AR  modeling)  will 
be  discussed  later.  Equation  (3,21)  can  be  written  as 


£yu 


iiuuj  ai 
L  J 


■a  ( o ) 


(3,22) 
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Now  consider  the  form  of  a  two  channel  autoregressive 

.w 

'  model  where  the  two  input  channels  are  y(k)  and  u(k) ;  that  is 


x^k)  =  y(k) 
x2(k)  =  u(k) 

Using  equations  (A. 4a)  and  (A, 7b)  the  two  channel  AR  modeling 
equations  are 
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and  comparing  aquations  (3.22)  and  (3.23)  it  is  clear  that 
with  the  exception  of  the  gain  term  a(0),  the  ARMA  modeling 
solution  oan  be  obtained  from  the  two  channel  AR  solution 
of  (3,23).  Furthermore  equation  (3.23)  can  bo  solved  inde¬ 
pendently  of  the  gain  term  via  the  Levinson  algorithm  as 
shown  in  Appendix  A  or  via  the  multichannel  lattice  methods 
developed  in  the  previous  chapter.  Then  all  that  remains 
to  complete  the  solution  for  the  ARMA  model  is  to  estimate 
the  gain  term  a(Q)  and  solve  tor  the  other  model  coeffiaienty 
using 
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-21 
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-a  ( 0 ) 


(3,24) 
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From  this  it  follows  that  the  transfer  functions  A(z)  and 
B(z)  for  the  ARMA  prediction  error  analysis  model  can  be 
related  to  the  two  channel  AR  prediction  error  matrix  poly¬ 
nomial  transfer  function  by 


[~  B(s) 


Cl  -  D(z)3 


-A(z) 


-a(  0) 


(3.25) 


It  is  also  easy  to  relate  the  ARMA  equation  error  to  the 
two  channel  AR  prediction  error  vector.  Using  equations 
(A. 3)  and  (A>5),  the  two  channel  AR  prediction  error  vector 
ia  written  as 


e(k)T  a 


y(k) 

u(k) 


C^k) L\  uOc)1] 


-11  -12 

-21  -22 


(3.26) 


Defining 


•*  Cey(k)  *u(k)2 


j  «a(Q) 


(3,27a) 


and  postmul  tip  lying  equation  (3.26)  by  $  and  using  (3.24)  it 
follows  that 

(3.27b) 

But  from  equation  (3,4a)  this  is  exactly  the  ARMA  model 
equation  error  so  that 

eQ(k)  =  e(k)T  £  (3.27c) 

and 

« 

e{eQ(k)2}  »  P  |  ( 3 . 27d) 

where  P  is  the  forward  prediction  error  covariance  matrix 
for  the  2  channel  AR  model.  Equation  (3,?7d)  also  provides 
a  means  of  estimating  the  gain  term  a(0)  after  the  two  channel 
AR  solution  has  been  obtained  by  setting  it  to  minimize  the 
mean  square  value  of  equation  error  resulting  in 

e £elt<k)«,,(k) ) 

a  (  0  )  *  . . . . (3,28) 

c  {«.  (k)^> 
u 

and  completing  the  ARMA  modal  solution, 


e(k)^  =  y(k)  -  a(0)u(k)  -  C%(k)T  I  u(k)*1']  •■»«- 
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The  portion  of  the  ARMA  model  solution  in  equation  (3.24) 
given  by  the  two  channel  AR  solution  can  be  found  recursively 
in  order  using  either  the  Levinson  algorithm  or  the  lattice 
filter  techniques,  If  the  desired  ARMA  model  order  is  not 
known  in  advance,  a  model  gain  term  a(Q)  can  be  estimated 
for  each  order  two  channel  AR  solution  to  find  the  ARMA 
model  of  corresponding  order  along  with  its  MSE.  In  this 
fashion,  the  entire  family  of  ARMA  models  for  the  system  from 
order  zero  to  order  N,  along  with  their  mean  square  errors 
are  obtained,  If,  on  the  other  hand,  the  desired  ARMA  model 
order  is  known  apriori,  the  gain  term  need  not  be  calculated 
at  each  stage,  Only  one  gain  term  must  be  calculated  to 
obtain  the  ARMA  solution  after  the  appropriate  order  two 
channel  AR  solution  has  been  found. 

It  has  already  been  shown  that  to  fully  identify  the 
system  using  the  equation  error  ARMA  formulation,  the  input 
signal  must  have  a  fiat  spectrum  as  in  the  case  of  white 
noise.  When  white  noise  is  used  as  the  system  input  u(k), 
simplifications  emerge  in  the  solution  of  the  two  channel 
AR  model  via  the  Levinson  algorithm  or  lattice  methods. 

For  a  white  sequence  with  variance  ou  , 


(n) 


0  ,  n  *  0 

au2  l  n  ■  0 


(3.29a) 
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0  ;  n  <  0 
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h(n)  ;  n  >  0 


(3.29b) 


where  h(n)  is  the  sampled  impulse  response  of  the  system. 
Consider  equation  (A, 16c)  for  . 
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(3.29c) 


This  shows  that  and  k!^  are  zero  and  furthermore  since 

/  -  \  /  n  \ 

r„„  and  r,,M  are  zero  for  all  n  it  is  seen  that 
— yu  — uu 


,<n> 

’22 


0 


( 3 . 29d) 


for  all  n  as  well,  This  aan  readily  be  understood  by  con¬ 
sidering  the  role  of  these  two  coefficients  at  eaah  stage 
in  the  AR  prediction  of  y(k)  and  u(k) ,  k^2  and  k22  are  the 

coefficients  used  in  trying  to  predict  u(k)  from  past  values 
of  y(k)  and  u(k) ,  and  when  u(k)  is  a  white  sequence  it 
oannot  be  predicted,  forcing  these  coefficients  to  be  zero. 
No  such  simplifications  oocur  in  the  backward  prediction 
problem  (and  therefore  in  the  matrioda)  since  even  for 
a  white  u(k),  a  backward  prediction  of  u(k-n)  from  subse¬ 
quent  values  of  u  and  y  is  possible,  This  is  because  in 


\ 
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general,  a  linear  dependence  of  y(k)  upon  past  and  present 
values  of  u(k)  can  occur  (and  certainly  will  occur  when  the 
relationship  between  y(k)  and  u(k)  is  described  by  an  ARMA 
model) , 

As  a  result  of  these  simplifications,  it  is  seen  from 
equation  (A. 19a)  that  the  polynomials  d^2(z)  and  d22(z)  are 
zero  when  u(k)  is  white  and  the  ARMA  model  is  given  by 


B(z) 


-A(z) 


l-duu> 


-a(0) 


(3.30) 


In  this  speaial  case  it  follows  that 


B(z)  =  detCl  -0  (z)1 


(3.31) 


and  therefore  stability  of  the  ARMA  model  and  of  the  two 
channel  AR  model  are  equivalent.  In  general,  however ,  no 
such  connection  exists  for  arbitrary  input  signals.  Further¬ 
more,  even  when  the  system  input  u(k)  is  white,  solutions 
for  k^2  and  k22  will  not  in  general  be  exactly  zero  sinoe 
the  required  correlations  are  usually  not  known  and  must  be 
estimated . 

This  development  showing  that  the  ARMA  model  solution 
can  be  obtained  from  a  two  channel  autoregressive  model  de~ 
pends  on  two  assumptions) 

1)  The  numerator  and  denominator  polynomial  orders  in 
the  model  transfer  function  are  assumed  to  be  the 


same  and  are  incremented  simultaneously  to  build  up 
the  desired  solution  recursively  in  order; 

2)  It  is  assumed  that  the  coefficient  of  z  to  the  zero 

power  (aCO))  in  the  numarator  polynomial  of  the  model  is 
either  known  in  advance,  or  that  another  means  of 
estimating  it  can  be  found  so  thnt  it  need  not  be 
estimated  directly  in  the  modeling  equations  of 
(3.20) , 

The  second  assumption  causes  no  concern  since  the  two  ohannel 

autoregressive  solution  is  obtained  independently  of  a(0), 

and  given  that  solution,  it  has  been  shown  that  a(0)  can 

indeed  be  estimated  in  another  fashion  in  equation  (3.28). 

« 

The  first  assumption  however,  warranta  further  consid¬ 
eration  since  it  seems  somewhat  restrictive  (at  first)  to 
require  that  the  numerator  and  denominator  polynomials  of 
the  model  have  the  same  order  when  in  faot,  the  system  being 
modeled  may  have  different  order,'  numerator  and  denominator 
polynomials,  To  see  that  this  assumption  is  not  restrictive 
in  general,  consider  what  1b  ocouring  as  the  model  is  built 
up  recursively  in  order.  At  eaoh  model  order  n,  the  pro¬ 
cedure  finds  the  best  n-th  order  model  (with  n  zeros  and  n 
poles)  in  a  minimum  mean  square  equation  error  sense. 

Making  the  model  numerator  and  denominator  orders  different 
(or  equivalently |  forcing  some  of  the  coefficients  to  zero 
in  the  model  where  the  orders  are  the  same)  pleats  a  priori 
constraints  on  the  model  *  forcing  some  of  the  poles  or 
zeros  to  the  origin  in  the  z»plane,  rather  than  allowing  the 
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model  to  place  them  at  will  to  minimize  the  cost  function. 

As  an  example,  consider  the  process  of  obtaining  an  ARMA 
model  for  a  system  given  by 

2 

2  d(n)  z“n 

H(z)  =  - 

1-  £  b(n)  z"n 
n=l 

where  two  of  the  ’systems  four  zeros  actually  occur  at  the 
origin  of  the  z  plane.  Constraining  any  of  the  model  zeros 
to  the  origin  at  orders  one ,  two  or  three  will  result  in  a 
model  with  higher  cost  (MSE)  than  if  they  were  not  con¬ 
strained,  Even  at  order  three,  a  model  without  constraints 
can  be  expected  to  use  the  "extra”  zero  to  help  in  approxi¬ 
mating  the  effects  of  the  system’s  fourth  pole  as  shown  in 
equation  (2.13)  yielding  a  lower  cost  and  more  accurate 
model  than  would  result  ’if  one  zero  were  forced  to  the  origin. 
Only  at  order  four  are  such  constraints  reasonable  but  even 
then,  they  are  not  necessary  since  the  modeling  procedure 
itself  should  recognize  that  the  best  fourth  order  model 
will  have  two  zeros  at  z-0. 

Therefore,  it  is  seen  that  assuming  equal  orders  for  the 
model  numerator  and  denominator  is  entirely  reasonable  as  a 
general  approach  in  obtaining  MMSE  models  for  unknown  sys¬ 
tems  or  even  reduced  order  models  for  known  systems.  When  it 
is  knowr  in  advance  that  the  best  MMSE  model  for  a  system 
has  zeros  at  z=0,  imposing  such  a  constraint  on  the  model  can 
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reduce  the  computational  complexity  of  obtaining  the  solution, 
but  even  here  the  assumption  of  equal  orders  is  not  restric¬ 
tive  . 

C,  LATTICE  FORM  ARMA  MODELING 

In  chapter  two  it  was  shown  that  the  lattice  structure 
of  Figure  2,7  could  be  applied  to  solve  the  multichannel 
AR  modeling  problem  in  terms  of  the  reflection  coefficient 
matrices  given  by  equations  (2.43).  Since  the  ARMA  model 
solution  can  be  obtained  from  a  two  channel  AR  solution  with 
y(k)  and  u(k)  as  the  input  channels,  only  the  structure 
described  by  equation  (3,27c)  need  be  added  to  a  two  channel 
AR  lattice  to  obtain  the  lattice  form  of  the  ARMA  analysis 
model,  It  is  interesting  to  look  at  the  exact  structure  of 
this  lattice  model  as  shown  in  Figure  3.4  for  a  second  order 
case,  This  structure  is  seen  as  a  lattice  interconnection 
of  two  single  channel  AR  lattices  operating  on  the  input 
signal. s  y(k)  and  u(k) ,  The  coefficients  on  the  main  diag¬ 
onals  of  the  K  and  K  matrices  specify  the  single  channel 
lattices  while  the  off  diagonal  elements  specify  the  inter¬ 
connections.  (This  will  also  be  the  case  for  extensions 
to  lattices  with  any  number  of  channels.) 

The  ARMA  synthesis  model  implementing  the  transfer  func¬ 
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tion  A(z)/B(z)  can  also  be  put  in  lattice  form,  The  for¬ 
ward  prediction  in  the  two  channel  AR  analysis  lattice  is 
described  by 


'.-"WWWSJF 
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®y<k)  ryCk)| 


eu(k)  eu'k) 


kll  k21  ey(k-l) 


k,  o  k„„i  e..(k-l) 


(3.32) 


and  the  equation  for  ek(k)^n+1^  can  be  rewritten  as 


ey(k)(n)  =  ey(k)(n+1,+k1'n+1)5y(k-l)(n)*k2<n+1,5u()c-l)Cn) 

(3.33) 


Equation  (3.33)  along  with  the  equation  of  eu  in  (3.32),  and 
the  equations  for  the  backward  prediction  (2.45b)  describe 

the  structure  shown  in  Figure  3 . 5  for  a  second  order  case . 

.  ( 2 ) 

To  provide  the  required  input  at  ey(k)  ,  recognize  from 

equation  (3.27c)  that 


e  (k)<2)  =  eQ(k)  +  a(0)eu(k)(2> 


(3.34a) 


If  the  ARMA  model  is  an  accurate  representation  of  the 
system,  the.  equation  error  eQ(k)  will  be  quite  small  (ideally 
zero)  so  that  in  general  for  the  N-th  order  case 


ey(k)(N)  *  a( 0)  eu(k)(N) 


(3.34b) 


This  is  indicated  by  the.  dashed  feedback  path  in  Figure  3.5. 
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D.  ARMA  MODEL  SIMULATIONS j  BATCH  PROCESSING 

ARMA  modeling  procedures  for  linear  systems  using  both 
the  lattice  filter  method  and  a  brute  force  matrix  inver¬ 
sion  with  equation  (3.20)  have  been  implemented  and  the 
results  of  these  two  approaches  have  been  compared  in  over 
a  thousand  model  simulations  of  more  than  thirty  different 
linear  systems.  The  experimental  results  which  follow  are 
a  representative  sampling  of  these  simulations. 

In  the  lattice  filter  method,  equations  (2.49)  were  used 
to  calculate  the  forward  and  backward  reflection  coefficient 
matrices,  with  time  averages  over  a  specified  interval  used 
to  estimate  the  required  correlations  in  and 

for  0  <■  n  _<  N-l.  Equations  (A. 15),  (A. 16a)  and  (A. 16b) 
were  used  to  obtain  the  two  channel  AR  model  coefficients 
from  the  K  and  K  matrices  .  and  with  the  gain  calculated 
in  equation  (3.28),  equation  (3.24)  was  used  to  obtain  the 
desired  ARMA  model  coefficients.  Equations  (A. 18)  were 
used  to  update  the  forward  and  backward  prediction  error 
covariance  matrices  from  one  lattice  stage  to  the  next. 
Figure  3.6a  provides  a  flow  diagram  of  the  procedure. 

In  the  brute  force  matrix  inversion  method  with  equation 
(3.20)  time  averaging  was  again  employed  to  estimate  the 
required  correlation  coefficients .  A  rectangular  window 
was  applied  to  the  data,  however,  to  retain  the  even  symmetry 
of  the  autocorrelation  function  in  these  estimates.  The 
ARMA  model  coefficients  were  then  obtained  using  a  general 
purpose  library  subroutine  (which  employed  gaussian 
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elimination)  to  solve  equation  (3.20).  Figure  3.6b  provides 
a  flow  diagram  of  this  procedure. 

In  both  cases  zero  mean,  unit  variance  gaussian  white 
noise  was  used  as  the  system  input.  In  the  simulation  re¬ 
sults  that  follow,  a  description  of  each  system  discussed 
(transfer  function  coefficients,  zero  locations  and  pole 
locations)  is  listed  in  tabular  form  and  root  locations  in 
the  z  plane  as  well  as  transfer  function  magnitudes  are 
plotted  for  the  system,  and  various  models  obtained  for  it. 

In  each  case,  models  were  obtained  for  averaging  intervals 
of  200,  500,  1000,  2000  and  4000  data  points.  Only  the 
results  for  the  two  extremes  of  200  and  4000  points  are 
included  here . 

The  first  system  considered  has asecond  order  numerator 
in  z  inverse  and  a  fourth  order  denominator,  and  its  char¬ 
acteristics  are  listed  in  Table  3.1.  Figure  3.7  shows  a 
comparison  of  the  root  locations  and  transfer  function 
magnitude  of  this  system  with  those  of  fourth  order  lattice 
filter  and  brute  force  models  obtained  when  correlations 
were  estimated  by  averaging  over  200  samples.  Figure  3.8 
provides  the  same  comparison  for  a  longer  averaging  interval 
of  4000  samples  of  data.  While  both  methods  perform  com¬ 
parably  with  the  longer  averaging  interval,  the  lattice 
method  produces  a  far  more  accurate  model  with  the  short 
averaging  interval.  It  is  also  interesting  to  compare  the 
performance  of  the  two  methods  when  the  system  is  overmodeled ; 
that  is ,  when  the  model  order  is  increased  beyond  that  of 
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Figure  3.6a.  Flow  diagram  of  batch  processing 
lattice  solution  for  all  ARMA 
models  orders  1  to  N . 
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.6b.  Flow  diagram  of  bate 
force  solution  for  a 
order  1  to  N. 


TABLE  3.1 


SYSTEM  A 

TRANSFER  FUNCTION  COEFFICIENTS 
NUMERATOR  DENOMINATOR 


AlO* 

0.25000 

A  Cl>* 

0.35000 

BID*  1.14000 

A  42)* 

0.24500 

5(2)*  - 1.45490 

A  (3)* 

0.0 

8(3)*  0.88490 

A  (41* 

0.0 

8(4)*  -0.40745 
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the  system.  Figures  3.9  and  3.10  show  such  a  comparison 
when  sixth  order  models  are  obtained  for  this  fourth  order 
system.  Ideally,  the  extra  zeros  and  poles  should  lie  at 
the  origin  in  the  z  plane  making  the  additional  transfer 
function  coefficients  zero.  Figure  3.9  shows  that  for  a 
200  point  averaging  interval,  the  lattice  locates  the  extra 
roots  in  the  vicinity  of  the  origin  while  the  brute  force 
model  does  not.  When  the  averaging  interval  is  increased 
to  4000  points,  the  extra  roots  of  the  lattice  model  move 
in  toward  the  origin  while  those  of  the  brute  force  model 
do  not.  Instead,  a  zero  pole  cancellation  at  some  arbitrary 
location  occurs  in  the  brute  force  model.  In  all  cases 
investigated  during  Jhis  effort,  the  lattice  method  clus¬ 
tered  the  extra  roots  in  the  vicinity  of  the  origin  and  as 
the  averaging  interval  was  increased  to  take  in  more  data, 
these  roots  were  consistently  moved  in  closer  to  the  origin. 
This  property  is  further  evidenced  by  the  plot  of  mean  square 
equation  error  as  a  function  of  model  order  shown  in  Figure 
3.11  for  a  200  point  averaging  interval.  The  MSE  for  the 
lattice  model  flattens  out  at  order  four  indicating  that 
further  increases  in  model  order  fail  to  increase  its 
accuracy.  Meanwhile  the  MSE  for  the  brute  force  model  con¬ 
tinues  to  decrease  beyond  fourth  order  as  it  uses  the  addi¬ 
tional  roots  to  reduce  modeling  errors  caused  by  innaccuracies 
in  the  fourth  order  model. 
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Figure  3,11. 


f«  and  brute  force  models  of 

a  ,,e+-«aim  k  with  200  point  averages. 
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To  investigate  the  ability  of  these  modeling  methods  to 
distinguish  roots  located  near  one  another  in  the  z  plane, 
a  pair  of  zeros  were  added  to  the  previous  system  in  close 
proximity  to  one  of  the  pole  pairs.  The  characteristics  of 
this  system  are  listed  in  Table  3.2.  Figures  3.12  and  3.13 
show  the  lattice  method  and  brute  force  modeling  results  for 
200  and  4000  point  averaging  intervals.  With  data  over  only 
200  sampling  instants,  neither  method  is  able  to  accurately 
model  the  effects  of  the  adjacent  roots.  When  the  averaging 
interval  is  increased,  the  lattice  correctly  models  the 
plant  while  the  brute  force  method  does  not,  and  even  results 
in  an  unstable  model.  (Figure  3.13b  comparing  the  transfer 
function  magnitudes  has  been  plotted  in  spite  of  the  model 
instability. ) 

To  investigate  the  ability  of  these  zero  pole  modeling 
methods  to  model  systems  that  are  actually  all  z.ero  or  all 
pole,  the  systems  listed  in  Tables  3.3  and  3.4  were  used. 

The  modeling  results  are  shown  in  Figures  3.14,  3.15,  3.16 
and  3.17, 

The  conclusions  drawn  from  the  results  of  this  experi¬ 
mental  study  are  as  follows; 

1)  For  short  data  lengths,  the  lattice  filter  method 
provides  more  accurate  models  than  does  the  brute 
force  modeling  Method. 

2)  When  the  system  is  overmodeled  using  the  lattice 
method,  the  excess  roots  are -clustered  about  the 
origin  in  the  z  plane  and  as  the  averaging  interval 
is  increased  and  more  data  taken  in,  these  roots  move 
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SYSTEM  B 

TRANSFER  FUNCTION  COEFFICIENTS 


NUMERATOR  DENOMINATOR 


A(C)« 

l.COOOO 

All)- 

1.34000 

8(  1)» 

1.14000 

A  (2)- 

1.79940 

8(2)- 

-1.A5490 

A  (2)- 

1.20A0C 

8(3)« 

C. 88490 

A  (A)  - 

0.88533 

B14)« 

-0.A0745 

ROCT  LOCATIONS 

ZEROS 

POLES 

RE 

IM 

RE 

IM 

7CC0C 

0.7000C 

0.50000 

0.5000C 

7CC0C 

-0.700CC 

0.50000 

-0.50000 

02000 

0.5500C 

0.07000 

c.soooc 

C2CC0 

-0.9500C 

0.07000 

-0.90000 
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SYSTEM  C 

TRANSFER  FUNCTION  COEFFICIENTS 
MJMERATOR  DENCMIN ATGR 

A  (C)  *  l.COCOO 

A<1)*  -1.50000  8(  1)«  C.C 

A  ( 2  )  •  C.62500  8<  2  J *  0.0 
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IM 

0.0 

0.0 
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RE  IM 

0.75000  0.25000 

C.75CCC  -0.2500C 


RE 

0.0 

0.0 


TABLE  3.4 
SYSTEM  0 

TRANSFER  FUNCTION  COEFFICIENTS 
NUMERATOR  DENOMINATOR 

A (Cl*  l.COOOO 

A  ( 1 1  *  0.0  B(l)>  1.50000 

A(2I*  C.O  8(2)*  -0.62500 
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RE  IM 

0.75000  0.25000 

0.75QQ0  -0.25000 


ZEROS 

RE  IM 

0.0  0.0 

Q.O  0.0 
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consistently  in  toward  the  origin.  This  also  forces 
the  excess  transfer  function  coefficients  and  re¬ 
flection  coefficients  to  be  very  small  (ideally  zero), 
clearly  indicating  that  the  model  order  is  higher 
than  necessary.  The  brute  force  method,  on  the  other 
hand,  scatters  the  excess  roots  throughout  the  z 
plane  and  produces  cancellations  of  the  excess  zeros 
and  poles.  This  results  in  nonzero  values  for  the 
excess  transfer  function  coefficients. 

3)  The  MSE  as  a  function  of  model  order  is  generally 
better  behaved  for  the  lattice  method  than  for  the 
brute  force  method,  decreasing  rapidly  until  the 
correct  model  order  is  reached  and  then  failing  to 
decrease  substantially  as  the  model  order  is  increased 
further. 

Two  qualitative  explanations  for  the  improved  performance 
of  the  lattice  filter  modeling  method  are  offered.  The 
first  is  that  the  data  is  not  windowed  in  the  lattice  method 
while  a  window  that  is  nonzero  only  over  the  span  of  the 
averaging  interval  must  be  used  in  the  brute  force  method  to 
maintain  even  symmetry  in  the  autocorrelation  function  esti¬ 
mates.  The  effects  of  this  difference  between  the  two  modeling 
methods  should  be  most  noticeable  for  short  data  lengths,  and 
become  less  evident  as  the  length  of  the  averaging  interval 
is  increased.  The  second  possible  cause  for  the  lattice 
method’s  better  performance  is  that  the  actual  output  se¬ 
quences  of  the  n-th  order  lattice  are  used  to  calculate  the 
coefficients  at  the  (n+l)-st  order  stage.  In  this  manner, 


modeling  errors  that  have  occurred  in  the  n-th  order  lattice 
can  be  compensated  for  to  some  extent  in  the  (n+l)-st  order 
stage.  No  similar  phenomenon  is  evident  in  the  brute  force 
modeling  approach.  Consider  the  differences  between  the 
Levinson  algorithm  (which  is  equivalent  to  the  matrix  inver¬ 
sion  method)  of  equations  (A. 16)  and  the  lattice  method  given 
by  equations  (2.49).  Both  methods  calculate  the  corrections 
that  must  be  made  to  the  n-th  order  model  to  obtain  the 
(n+l)-st  order  model  in  terms  of  K^n+1^  and  and 

theoretically, 

e{e<n)(k)  e<n)(k-l)T>  =  R  (n+l)T  -  P  (n)T  ]0fn) 

• 

When  correlations  are  estimated  by  averaging  over  finite 
intervals  however  this  equality  will  not  in  general  be 
satisfied  making  the  two  methods  different.  The  Levinson 
algorithm  will  estimate  the  correction  terms  to  be  added  to 
the  optimum  n-th  order  model  while  the  lattice  method  esti¬ 
mates  the  correction  terms  to  be  added  to  the  estimated  n-;th 
order  model  actually  obtained. 

The  improved  performance  of  the  lattice  method  is  not 
achieved  without  cost,  however.  The  method  is  made  compu¬ 
tationally  expensive  by  the  need  to  store  the  system  input 
and  output  sequences  and  the  lattice  prediction  error  se¬ 
quences  and  pass  them  through  successive  stages  of  the 
lattice  as  it  is  built  up  in  order.  The  computational  com¬ 
plexity  of  the  two  methods  is  compared  in  Table  3.5. 
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Table  3,5.  Computational  requirements  for  batch 
processing  ARMA  modeling  of  an  N-th 
order  system  using  P  samples  of  the 
system  input  and  output. 


Number  of 
Correlat ion 
Estimates 
Required 


4N+3 


Matrix 

Inversions 


1  - 


Data  Storage 
Requirements 


Computations 
To  Pass  Data 
Through  The 
Filter 


averaged  over 
P  data  points 


4N+3 


dimension  2N+1 


2N -dimens ion  2 
1-dimension  1 


N .  A. 


2P  samples 


8NP  multiplica 
tions 

4NP  additions 


N .  A. 


E.  ADAPTIVE  LATTICE  ARMA  MODELING 
O 

In  addition  to  the  batch  processing  method  described  in 
the  previous  section,  the  lattice  ARMA  analysis  model  can  be 
implemented  adaptively  as  well.  The  adaptive  lattice  solu¬ 
tion  for  the  mulxichannel  AR  lattice,  which  solves  most  of 
the  ARMA  modeling  problem,  has  already  been  described  in 
chapter  II.  To  make  the  lattice  ARMA  model  adaptive,  only 
an  adaptive  solution  for  the  gain  term  a(0)  need  be  added. 

To  avoid  ambiguity,  the  time  varying  adaptive  estimate  of 
this  term  at  time  k  is  denoted  by  aQ(k) .  Applying  an  LMS 
adaptive  algorithm  it  follows  that 

aQ(k+l)  =  aQ(k)  -  yQ  V(k)  (3.35) 

and  using  equation  (3.27d)  to  form  an  instantaneous  .stimate 
of  the  gradient  yields 

A 

V(k)  =  -2  eu(k)Cey(k)-aQ(k)eu(k)]  (3.36) 

=  -2  eM(k)  e  (k) 
u  o 

so  that 

aQ(k+l)  =  aQ(k)  +  2uQ  eQ(k)  eQ(k)  (3.37) 

*  ► 
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Here  it  is  clear  that 

1)  eQ(k)  is  analagous  to  the  error  signal. 

2)  eu'k)  is  analagous  to  the  input  signal, 

3)  e  (k)  is  analagous  to  the  desired  signal. 

*/ 

so  that  for  stability  y^  must  satisfy 


0  <  y  <  - - — 9- 

0  e{eu(k)^} 


(3.38) 


Once  again,  however ,  the  mean  square  value  of  eu(k)  may  vary 
from  stage  to  stage  and  over  time  as  the  two  channel  AR 
lattice  adapts  making  it  appropriate  to  apply  relations 
similar  to  equations  (2.58)  and  (2.59) 

U0  <k)  S  oJJTKT  (3.39a) 

aQ  (k)  =  (1-a)  aQ  (k-1)  +  a  eu(k)2  (3.39b) 

where  a  is  the  normalized  adaptive  step  size  and  the  depen¬ 
dence  on  order  is  implicit  (a  superscript  (n)  could  be  used 
on  aQ,  aQ,  V,  and  all  the  en  -r  terms  in  equations  (3.35) 
through  (3.39)  to  explicitly  denote  their  dependence  on  the 
order  of  the  solution) , 

This  adaptive  lattice  ARMA  modeling  scheme  was  implemen¬ 
ted  and  the  results  of  its  use  in  modeling  system  A  described 
in  Table  3.1  are  presented  here.  A  normalized  adaptive  step 
size  of  a  *  .05  was  used  in  the  following  simulations  and 
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the  results  represent  an  ensemble  average  of  one  hundred 
trials.  Unity  variance  white  noise  was  used  as  the  system 
input,  A  flow  diagram  of  the  procedure  is  shown  in  Figure 
3.13. 

Figure  3,19  shows  a  plot  of  the  mean  square  equation 

error  as  a  function  of  time  for  a  fourth  order  model  while 

it  adapts  and  Figure  3,20  shows  the  behavior  of  one  term  in 

the  K  matrix,  k£^  ,  and  one  term  in  the  £  matrix,  k^}  at 

the  first  five  lattice  stages,  1  <  n  <  5 .  These  graphs  of 

the  k11  terms  clearly  show  the  successive  stage  by  stage 

manner  in  which  the  lattice  model  adapts.  Figure  3.21  shows 

« 

a  comparison  of  the  transfer  function  magnitude  and  root 
locations  of  the  system  with  those  of  the  fourth  order 
adaptive  model  after  1000  and  2000  iterations.  Figure  3.22 
shows  the  same  comparison  in  the  overpodeled  case  when  a 
sixth  order  model  is  obtained  for  this  fourth  order  system. 

While  these  results  show  that  tne  adaptive  solution 
performs  well  and  is  a  viable  alternative  to  the  batch  pro¬ 
cessing  solution,  Figures  3.22c  and  3. 2 2d  show  that  one 
advantageous  characteristic  of  the  batch  solution  has  not 
carried  over.  The  excess  roots  in  the  overmodeled  case  are 
not  tightly  clustered  in  the  vicinity  of  the  origin  indi¬ 
cating  that  the  excess  transfer  function  coefficients  are  not 
near  zero.  In  examining  Figure  3.20  it  is  also  evident  that 
the  convergence  of  the  reflection  coefficients  at  che 
overmodeled  lattice  stage  (k^ip  and  towards  their  true 

value  of  zero  is  qui.te  slow.  This  relatively  slow  tracking 


Initialize  the  parameter 
and  step  size  normalizing 
factors 


Sample  system  input 
and  output 


Find  current  value  of  ev  '(n) 
and  e^n^(k)  for  0  <  n  <  N 


Update  the  step  size 
normalizing  factors 
using  equations  (2.59) 


Update  the  K(n)  and  K(n) 
estimates  using  equations 
(2.55) 

and  (2.58)  for  1  <  n  <  N 


Update  a~  using  equations 
(3.37)  and  (3.39a)  for  n 
equal  to  the  desired  order 
ARMA  solution 


If  desired,  calculate_ARMA 
coefficients  from  K,  K  and  a, 
estimates  using  equations  1 
(A. 15) ,  (A. 16)  and  (3.24). 
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Figure  3.19 


Mean  square  value  of  equation  error  for  the 
fourth  order  adaptive  lattice  model  as  a 
function  of  time. 
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+  “  of  overmodeled,  zero  valued  parameters  has  been  found  to  be 

a  general  characteristic  of  the  adaptive  lattice  algorithm 
and  has  also  been  noted  briefly  by  Morf.  [Ref.  38] 

To  understand  the  reason  for  this  behavior,  consider 
what  occurs  as  the  overmodeled  fifth  stage  in  the  previous 
example  adapts.  Initially,  before  the  coefficients  of  the 
first  four  lattice  stages  converge  to  their  optimum  values, 
the  prediction  error  sequences  out  of  the  fourth  stage  are 
large  and  suboptimum.  These  signals  provide  incorrect  in¬ 
puts  to  the  fifth  stage  driving  its  parameter  estimates  to 
some  values  other  than  their  optimum  zero  values.  As  the 
first  four  stages  converge,  the  prediction  error  sequences 
going  into  the  fifth  stage  get  small  and  since  they  drive 
the  gradient  estimates ,  convergence  back  toward  zero  is 
quite  slow  in  these  parameter  estimates. 

The  cost  functions  being  minimized  at  the  overmodeled 

(  c  )  ,(  c  \ 

fifth  stage  are  the  trace  of  P  and  F  given  by 

p(5)  ,  pC4)  .  s<5)  .  +  k(5)TeC“)k‘5> 

(3.40a) 

F<5>  =  F<“>  .  A«‘*>VS)  -  7<s)V4)  *  K,5)V4¥5> 

(3.40b) 

Applying  the  results  of  Appendix  F,  the  parabolic  surfaces 

defined  by  these  cost  functions  are  described  by  the  eigen- 

values  and  eigenvectors  of  P  and  P  ,  the  prediction  error 
**  1  ~  ~ 

covariance  matrices  at  the  fourth  stage.  Consider  the  forward 
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predictions.  The  actual  system  output  is  given  by 


y(k)  =.  T]  b(i)  y(k-i)  +  £  a(i)  u(k-i)  (3,41) 

L=1  i=0 

and  for  a  white  input  signal,  the  minimum  errors  in  the 
fourth  order  two  channel  autoregressive  predictions  of  y  and 
u  are 


<4>  (k)  =  a(0)  u(k) 
.<4)  (k)  =  u(k) 


C  3.42a) 


(3.42b) 


This  results  in  an  optimal  prediction  error  covariance  matrix 
given  by 


a(0 ) 2 
a(  0 ) 


a(0) 


Ruu(0) 


(3.43) 


2 

with  eigenvalues  of  0  and  l+a(0)  ,  Also  in  the  case  of  the 
system  described  by  Table  3,1  where  a(3)  =  a(4)  =  0,  it  is 
seen  from  equation  (3.41)  that  a  perfect  backward  two  channel 
AR  prediction  of  y(k-4)  is  possible  resulting  in  an  optimal 
backward  prediction  error  covariance  matrix  of 


U 

0 


e{i<4)(k>) 


(3.44) 
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which  also  has  a  zero  eigenvalue . 

Since  the  ellipses  obtained  by  passing  a  plane  through 
the  parabolic  cost  surface  have  axes  whose  half  lengths  given 
by  1/AT  and  since  P  and  P  each  have  one  eigenvalue  of 
zero,  it  is  seen  that  the  parabolic  bowls  along  which 

/  M 

and  1C  adapt,  degenerate  toward  infinitely  long  parabolic 
troughs  as  the  first  four  stages  converge  toward  their  opti¬ 
mum  values.  This  is  responsible  for  the  slow  convergence  of 
the  overmodeled  parameters  back  toward  zero.  To  avoid  this 
problem,  some  means  of  detecting  this  degeneration  of  the 
cost  surface  and  then  reseting  the  appropriate  parameters 
to  zero  must  be  found  and  this  certainly  provides  an  inter¬ 
esting  area  for  future  study. 


F.  A  LATTICE  APPROACH  FOR  MULTICH/WNEL  ARMA  MODELING 


The  lattice  filter  solution  methods  for  the  ARMA  model 
can  readily  be  generalized  to  multiple  input  multiple  output 
ARMA  models  for  linear  systems.  The  equations  for  the 
multichannel  ARMA  model  of  the  system  shown  in  Figure  3.21 
are  developed  in  Appendix  G  with  the  solution  for  the  model 
coefficients  given  by  equation  (G.7)  repeated  here  for 
convenience . 
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Figure  3.21.  A  general  multichannel  ARMA  system 


This  is  clearly  a  generalization  of  the  single  channel  ARMA 

equation  error  solution  given  by  equation  (3.20)  and  just 

as  in  the  single  channel  case  some  assumptions  are  required 

to  apply  a  Levinson  type  algorithm. 

If  it  is  assumed  that  all  the  a..(0)  are  known  or  can 

^  J 

be  estimated  in  another  manner,  they  can  be  incorporated 
into  a  matrix  given  by 


^0 


a^CO) 


an  .(0).#*. 

Qil 


(0) 


(3.46) 


and  equation  (3.45)  becomes 
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This  is  similar  in  form  to  the  equations  for  the  solution  of 
a  (Q^  +  Qq)  channel  autoregression  with  input  channels  y^(k) , 

. . . ,  yn  (k) ,  u, (k) ,  . , . ,  un  (k)  so  that  the  multichannel 
ARMA  solution  is  related  to  the  multichannel  AR  solution  by 


C3.48) 


The  multichannel  AR  prediction  error  vector  is  given  by 


e(k)T  = 


Z  OO 

T 

iv(k) 

u  ( k ) 

mm  mm 

r  T  1 

-  [Y1  1 

“  1 

uT]  D  * 

y 

and  defining 


(3.49) 


(3.50) 


it  follows  that 

e(k)T£  *  y;(k)T  -  u(k)TAQ 


[YJ 


UA] 


(3.51) 


=  eQ(k) 


T 
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establishing  the  generalization  of  equation  3.27c  relating 
the  multichannel  AR  prediction  error  vector  to  the  multi¬ 
channel  ARMA  er:or  vector  e^k).  The  ARMA  prediction  error 
covariance  then  is  given  by 


PQ  s  £'  P  ± 


(3.52) 


and  the  coefficient  matrix  AQ  can  be  set  to  minimize  the 
trace  of  Pn  resulting  in  a  solution  given  by 


A n  =  e{e  (k)  e  (k)  } 
— u  — u  — u 


-1 


e{eu(k)  ey(k)  ) 


(3.53) 


completing  the  multichannel  generalization  of  the  single 
channel  results .  The  portion  of  the  solution  given  by  the 
multichannel  autoregression  can  be  solved  as  before  using 
the  lattice  methods  in  either  batch  or  adaptive  fashion. 

Then  the  matrix  of  gains  can  be  obtained  from  equation  (3.53) 
by  batch  processing  or  equations  (3.37)  and  (3.39)  can  be 
generalized  to  yield  an  adaptive  solution  given  by 


AqU+I) 


AQ(k)  +  2 


a 

^ao 


eu(k)  eQ(k)‘ 


(3.54a) 


where 


Og(k)  =  (1-q)  Cg(k-l)  +  q  eu(k)T  eu(k) 


(3,54b) 


( 


fc. 


15D 


t  r 


It  is  clear  that  the  equations  and  methods  developed  earlier 
for  the  single  channel  ARMA  model  are  a  special  case  of 
these  results  with  Q.  =  Qn  = 
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IV.  NONLINEAR  SYSTEM  MODELING 


The  modeling  of  nonlinear  systems  is  a  far  more  complex 
problem  than  linear  systems  modeling.  No  attempt  is  made 
here  to  provide  a  comprehensive  treatment  of  the  problem. 
Rather,  two  specific  models  for  systems  comprised  of  the 
interconnection  of  linear  and  memoryless  nonlinear  subsys¬ 
tems  are  considered.  Both  of  these  models,  the  Volterra 
model  and  the  new  nonlinear  ARMA  model,  are  shown  to  be 
generalizations  of  the  MA  and  ARMA  modeling  problems  explored 
previously  so  that  with  appropriate  modifications,  the 
Levinson  algorithm  and  lattice  methods  can  be  used  to  solve 
for  the  model  parameters. 

A.  VOLTERRA  NONLINEAR  MODELING 

The  Volterra  series  model  characterizes  nonlinear  sys¬ 
tems  using  a  generalization  of  convolution  where  the  system 
output  is  approximated  as  a  summation  (possibly  infinite) 
of  outputs  of  degree  m  systems. 

*  M 

y(k)  =  £  ym(k)  (4.1a) 

m»l 

This  is  shown  pictoriaily  in  Figure  4.1, 
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Figure  4.1.  The  Volterra  model  for  nonlinear  systems 


A  number  of  representations  for  these  m-th  degree  systems 
are  possible.  The  most  commonly  used  representation  is  in 
terms  of  symmetric  Volterra  kernels  and  is  given  by 


y_(k) 

in 


*  £••••£ 


n  =0 
m 


nl=0 


a  (n, ...n  )u(k-n, ) . . .u(k-n  )  (4.1b) 

s  1  m  1  m 


and  as^n]_*‘*nm^  the  m-th  degree  symmetric  Volterra  kernel, 
(Any  permutation  of  the  indicies  results  in  the  same  value 
for  this  kernel  giving  a  high  degree  of  symmetry.) 

This  model  arises  quite  naturally  for  a  linear  system  in 
cascade  with  a  power  series  nonlinearity  as  shown  in  Figure 
4.2  for  a  quadratic  nonlinearity  where  the  output  can  be 
written  as 


153 


( 


00 


00 


(4.2a) 


y(k) 


£  ^  a(n1)a(n2)u(k-n1)u(k-n2) 

n^O  n2=0 


and 


as^nln2 )  S  a^ni^a^n2^ 


(4.2b) 


u(k) 


y(k) 


Figure  4.2.  A  quadratic  nonlinear  system. 


The  Volterra  series  model  in  this  form  has  been  widely  dis¬ 
cussed  in  the  literature  [e.g.  Ref.  1,  6,  13,  14,  25,  26  ,  48 
and  54]  since  Wiener  [Ref.  62]  first  applied  it  to  systems 
analysis  and  modeling.  It  has  the  added  benefit  of  treating 
linear  systems  as  a  special  case  of  the  model  since  the 
first  degree  kernel  is  exactly  the  convolutional  represen¬ 
tation  of  the  MA  model. 

The  number  of  kernels  (M)  required  for  an  accurate  model 
depends  on  the  nature  of  the  nonlinearity  in  the  system  and  as 
long  as  the  nonlinearity  is  soft,  a  relatively  low  degree 
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model  will  suffice,  keeping  the  problem  manageable  in  that 
regard.  The  primary  difficulty  associated  with  Volterra 
nonlinear  modeling  arises  from  the  fact  that  it  uses  a 
nonrecursive  MA  representation  for  the  linear  portion  of 
the  system,  requiring  in  general,  an  infinite  memory  as 
indicated  by  the  upper  limits  on  the  summations  in  equation 
(4.1b).  In  practice,  these  summations  need  to  be  truncated 
as  shown  in  equation  (4.3) 


y_(k) 

m 


N 


N 


E  •••£ 

V°  V° 


a  (n, . . .n  >u(k-n, > . . .u(k-n  )  (4.3) 

slm  l  m 


but  a  large  number  of  terms  may  still  be  required  to 
accurately  model  the  system.  One  method  of  solving  for  the 
model  is  to  set  the  parameters  (terms  of  the  Volterra  kernels) 
to  obtain  a  minimum  mean  square  equation  error  where  the 
equation  error  is  defined  as  the  difference  between  the  sys- 

A 

tern  output  and  y(k) . 

To  simplify  the  solution  and  reduce  the  number  of  para¬ 
meters  that  must  be  obtained,  the  symmetry  of  the  kernels  can 
be  exploited  by  rewriting  equation  (4,1b)  as 


ymoo 


£  . . .  £  a+(n.  , .  .„n  )u(k-n,) . .  ,u(k-n  ) 

n,  =0  n0  =  n,  n  =n_  ,  i 

I  i  l  m  m-i 

(4.4) 


where  at(n1 . . .nm)  is  the  m-th  degree  triangular  Volterra 
kernel.  For  a  finite  upper  limit  of  N  on  the  summations 
the  m-th  degree  symmetric  kernel  will  contain  (N+l)m  terms 


155 


of  them  are  distinct  with  the  remainder  de- 


v..*. _ (N+m)! 

but  only  -^7-57 

termined  by  symmetry  considerations. 

Still  another  representation  for  ym(k)  uses  the  regular 

form  of  the  Volterra  kernel  (this  terminology  has  recently 

been  introduced  by  Mitzel* Clancy  and  Rugh  [Refs.  7  and  35]). 

It  is  given  by 

00  00 

y_(k)  =  £  ...  £  a  (h, . . .h_)u(k-h1 )u(k-h, -h„) 

m  h  =0  h  =0  r  1  m  1  12 

ll  m 

...  u(k-h, -. . .-h_)  (4.5) 

1  m 

where  ar(h^...hm)  is  the  m-th  degree  regular  Volterra  kernel. 
With  infinite  upper  limits  on  the  summations,  the  symmetric, 
triangular  and  regular  forms  of  the  Volterra  kernels  are 
equivalent,  however,  when  finite  upper  limits  are  used  they 
cover  the  field  of  the  model  kernels  in  different  ways. 

Because  of  its  symmetry,  it  is  reasonable  to  have  equal 
upper  limits  on  all  the  summations  on  the  symmetric  kernel 
as  was  done  in  equation  (4.3).  This  is  shown  in  Figure  4.3a 
for  a  second  degree  kernel.  The  equivalent  triangular  kernel 
is  given  by 


N  N 

y  (k)  n  £  •  •  •  £  a4.(n1  . ,  .n_)u(k-n. ) , .  .u(k-n)  (4,6a) 

■'m  t  1  m  l  m 

n, =C  n  =n  , 
l  m  m**x 

and  it  covers  the  region  shown  in  Figure  4.3b  for  a  second 
degree  case.  The  corresponding  regular  form  expansion,  however , 
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requires  variable  upper  limits  on  the  summations  to  cover 
the  equivalent  kernel  space  as  shown  in  equation  (4.6b),  and 
Figure  4.3c  for  a  second  degree  kernel. 


y_(k) 

IT. 


N  N-h,  N-h 

=  E  E  -  E 

V°  V°  V° 


a  (h,  . .  .h  )u(k-h,  )u(k~h,  -h,,) 
rim  x  I  a 


• . , u(k-h. -. . . -h  ) 
x  m 


(4.6b) 


Thus  it  is  seen  that  only  half  the  field  needs  to  be  covered 
by  the  triangular  and  regular  expansions  to  identify  the 
kernel  associated  with  the  square  field  of  the  regular  ex¬ 
pansion.  When  the  regular  form  expansion  is  used  with  con¬ 
stant  finite  upper  limits  there  is  no  inherent  reason  to 
make  all  the  upper  limits  equal  since  there  is  no  symmetry 
in  the  kernel.  Considering  the  regular  expansion 

N,  Nm 

y_(k)  =  . .  .  T)  a  (h. . . .h  )u(k-h, ) . . .  (u(k~h, -. < .-h  ) 

m  w  an  w  =n  r  1  m  1  1  m 


h.=0  h=0 

l  m 


a  triangular  expansion  given  by 


N,  h,+N„  h_  ,+N. 


(4.7a) 


i  1^,  2  ‘  W  m 

y0(k)  =  zZ  2*  •  •  •  2J  a<.(n,  .  .  .n  )u(k-n, ) . .  .u(k-n  ) 

2  „  =  n  t  1  m  1  m 


n,  =0  n,,=n.  n  =nm  , 
l  z  l  m  m-i 


(4.7b) 


is  required  to  cover  the  corresponding  field.  For  a  quadratic 
expansion  this  is  shown  in  Figures  4.4c  and  4.4b,  The  equi¬ 
valent  region  in  the  symmetric  field  is  shown  in  Figure  4.2a. 
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Therefore,  identifying  a  rectangular  kernel  in  the  regular 
kernel  snace  is  equivalent  to  obtaining  a  symmetric  kernel 
in  the  arrow  shaped  region  of  Figure  4.4a. 

Which  type  of  expansion  is  more  appropriate  for  a  given 
system  depends  on  the  shape  of  the  kernels  for  that  system. 
For  example,  if  a  quadratic  nonlinear  system  has  a  kernel 
with  a  relatively  square  shape  in  the  symmetric  kernel 
space,  a  regular  form  expansion  with  constant  upper  summa¬ 
tion  limits  will  have  to  estimate  many  zero  valued  terms 
and  is  inefficient.  On  the  other  hand  if  the  system  has  a 
kernel  similar  to  the  arrow  shaped  region  of  Figure  4.4a 
in  the  symmetric  space,  a  regular  expansion  will  be  efficient 
while  a  symmetric  expansion  over  a  larger  field  would  be 
required  with  many  zero  valued  parameters.  Not  enough  is 
known  about  how  to  relate  types  of  nonlinear  systems  with 
various  shaped  kernels  however,  and  the  question  of  kernel 
shape  and  the  best  type  of  expansion  is  not  pursued  further 
here . 


1 .  Lattice  Filter  Methods  For  Volterra  Modeling 

Lattice  filter  methods  derived  earlier  can  be  applied 
to  obtain  a  minimum  mean  square  equation  error  solution  for 
the  Volterra  model  if  the  regular  form  of  the  expansion  is 
used.  The  Volterra  model  can  be  put  in  the  form  of  a  mul¬ 
tiple  input  single  output  MA  model  by  defining  a  new  family 
of  signals  as  nonlinear  combinations  of  delayed  values  of 
the  system  input  u(k). 
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u. 


2 


h  (k)=u(k)  u(k-h2)  u(k-h2-h3 ) , . .uCk-hj- • . . -h^) 
m 

(4.8) 


For  finite  summations  the  regular  form  of  the  expansion 
becomes 


Nm2 

ym<*>  2  £  • • *  £ 

m  h  =0  hT=0 

m  2 


52  a  (h,  ...h  )u.  ....  (k-h, ) 

,  r  1  m  h0  n_  1 

h^-Q  z  m 


(4.9) 


and  can  be  regarded  as  the  sum  of  the  outputs  of  a  large 

number  of  linear  filters  whose  inputs  are  given  by  u^  .  .*h  (k). 

2  m 

Equation  (4.9)  is  exactly  the  form  of  a  Q.  input,  single 
output  MA  model  where 


Q.  =  (N9+1)(N,+1). .. (N  +1)  (4.10) 

i  mz  mo  mm 

Furthermore,  if  the  same  upper  limit  on  the  summations  over 
h^  is  used  in  each  of  the  various  m-th  degree  systems , 
^NllsN21s '  ‘  *  sNml^  *  overa-1-^  Volterra  model  given  by 
equations  (4.1a)  and  (4.9)  is  in  a  form  suitable  for  solu¬ 
tion  via  the  multichannel  Levinson  algorithm  or  the  multi¬ 
channel  MA  lattice  methods.  The  requirement  to  use  the  same 
upper  limit  on  all  summations  over  h^  arises  because  the 
Levinson  algorithm  and  lattice  methods  assume  the  same  amount 
of  memory  in  each  channel  of  the  model, 

As  a  specific  example  consider  a  second  degree  ex¬ 
pansion  where  nj_is^21s^1  anc^  N21s^2* 
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n2  nx 


y(k)  =  £  a  (h. )u(k-h. )  +  S  2  a  (h, h0 )u.  (k-h, ) 


h1=0 


h2=0  hx=0 


(4.11a) 


(k)  =  u(k)u(k-h2> 


(4.11b) 


Defining  data  and  coefficient  vectors  for  each  channel  given 
by 


ut  (k)  =  [u.  (k)...u.  (k-N,  )]T 

* *ri  ^  1*2  Hi  2  • 


(4.12a) 


Sh. 


Car(0 ,h2> • . •ar(N1,  h2)]' 


(4.12b) 


and  embedding  them  into  single  data  and  coefficient  vectors 
written  as 


X+(k)  =  Cu+(k)^  j  ut(k)T  i  ...  i  uj  (k)T]  (4.12c) 

—  “  i  “°  I  i  ~^2 


+T  I  +T 

*  La 


-0 


I  I  +T  T 

|  •  •  •  |.  J 
I  i  2 


( 4 . 12d) 


the  equation  for  the  Volterra  model  output  becomes 


y(k)  =  X*  (k)T  d+ 


(4 . 12e) 
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which  is  clearly  of  the  same  form  as  equation  (A. 24a)  and 
represents  a  ^  +  2  input,  single  output  MA  model.  All  the 
nonlinearities  in  the  Volterra  model  are  external  to  this 
MA  model,  in  the  formation  of  its  various  input  signals  from 
the  system  input  u(k) .  This  model  is  illustrated  in  Figure 
4.5. 

It  is  interesting  to  consider  what  the  recursive  in 
order  nature  of  the  Levinson  algorithm  or  lattice  methods 
mean  to  the  nonlinear  Volterra  problem.  In  building  up  the 
MA  model  solution  recursively  in  order,  the  upper  limit  cf 
the  summations  over  h^  is  increased  until  the  desired  value 
is  reached.  In  terms  of  the  Volterra  model  kernels,  this 
means  allowing  each  of  the  regular  form  kernels  to  grow  in 
size  in  the  h^  dimension  while  holding  their  boundaries  fixed 
at  prespecified  values  in  all  the  other  dimensions.  In  the 
linear  MA  model,  the  kernel  has  only  one  dimension  and  there¬ 
fore  the  recursive  in  order  solution  eliminates  any  require¬ 
ment  to  prespecify  its  upper  limit  (the  order  of  the  model). 
For  the  higher  degree  nonlinear  kernels,  these- methods  reduce 
by  one  the  number  of  kernel  boundaries  that  must  be  pre¬ 
specified  for  each  kernel.  Allowing  the  kernel  to  grow  in 
any  of  the  other  M-l  dimensions  (hg  through  h^)  corresponds 
to  adding  additional  channels  to  an  existing  lattice  and 
simple  methods  of  accomplishing  this  are  not  available. 


Figure  4,5.  A  second  degree  nonlinear  Volterra 
model  in  multichannel  MA  form. 

The  represents  are  coeffi- 
cients^of  the  single  input  single 
output  MA  models  shown. 
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3.  NONLINEAR  ARMA  MODELING 

As  was  previously  mentioned,  the  primary  difficulty 
associated  with  the  Volterra  series  model  arises  from  the 
fact  that  it  is  a  nonlinear  generalization  of  the  MA  model 
and  as  such,  a  large  number  of  terns  may  be  required  to 
accurately  represent  even  a  mildly  nonlinear  system.  In 
linear  system  modeling  this  difficulty  was  remedied  by 
using  the  more  general  ARMA  model.  It  is  reasonable  to 
assume  therefore  that  a  nonlinear  generalization  of  the 
ARMA  model  could  remedy  the  problem  in  the  nonlinear 
modeling  case  (for  at  least  certain  types  of  nonlinear 
systems).  Such  a  generalization  called  the  nonlinear  ARMA 
model  has  recently  been  proposed.  [Ref.  64]  This  model 
forms  an  estimate  of  the  current  value  of  the  system  output 
as  follows: 

OO  09 

A  QO  _ 

y(k)  =  E  a  (n.  )u(k-n. )  ♦  23  £  a  (n.n-,)u(k-n.  )u(k-n») 
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^  ^ ^ ^  C(n.m..  )u(k-n.  )  y  ( k— m*.  )  ^  •  •  •  ^  2j * ••  )C  ...  ^  •  •  • 

1^=0  m1=l  1  x  1  1  n^O  np=0  m.=l 

00 

...  2  C(n^ . . .n^m^ . .  .itiq)u(k-n^) . .  u  ( k-n^ )  y  (k-m^ ) 

mq=i 

. . .y(k-M  )  (4.13) 

q 

The  first  three  terms  of  equation  (4.13)  are  a  discrete 
Volterra  expansion  of  the  input  signal  u(k)  and  represent 
F^gCuOO]  in  terms  of  the  discussion  of  Chapter  I.  The 
second  three  terms  are  a  discrete  Volterra  expansion  on  the 
system  output  delayed  one  sample  interval  and  represent 
F20Cy(k-l)].  The  final  two  terms  are  bivariate  expansions 
of  the  system  input  and  delayed  output  and  represent 
F3Q[u(k),  y(k-l)3.  (This  is  the  first  and  only  model  con¬ 
sidered  where  F^gO ]  is  not  assumed  to  be  zero.)  Equation 
(4.13)  is  clearly  a  nonlinear  extension  of  the  linear  ARMA 
model  contained  in  the  first  and  fourth  terms.  As  was  the 
case  in  the  Volterra  model*  the  number  of  multiple  summations 
required  in  (4.13)  is  dependent  upon  the  nature  of  the  non¬ 
linearity  in  the  system  being  modeled.  The  upper  limits  on 
the  summations  in  (4.13)  have  been  written  as  infinity  in¬ 
dicating  a  requirement  for  infinite  memory  or  model  order. 

As  is  discussed  subsequently*  however,  the  required  model 
order  (memory)  may  in  f^ct  be  finite  due  to  the  nature  of 
the  system  being  modeled,  thereby  alleviating  the  difficulty 
encountered  in  the  Volterra  nonlinear  model  previously  presented. 
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The  kernels  of  the  input  expansion  and  output  expansion 
as<*)  and  bg(*)  are  symmetric  since  any  permutation  of  the 
indicies  results  in  the  same  value  for  the  kernel.  They 
have  therefore  been  labeled  with  a  subscript  "s"  and  can 
also  be  written  in  triangular  and  regular  form. 
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(4.15) 


In  writing  the  triangular  and  regular  forms  in  equation  (4.15) 
the  lower  index  on  the  summations  has  been  shifted  to  zero. 

In  the  case  of  the  bivariate  expansion  terms  in  equation 
(4.13)  the  kernels  do  not  possess  symmetry  so  that  trian¬ 
gular  expansions  are  not  possible  but  regular  form  expansions 
are  possible. 
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(4.16) 

Thus  two  regular  forms  are  possible  for  the  bivariate  expan¬ 
sions.  Figures  4.6  and  4.7  illustrate  the  manner  in  which 
the  regular  form  of  the  bivariate  expansion  covers  the  field 
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of  model  kernels  for  a  second  degree  case  for  finite  upper 
limits  of  and  Ng  on  the  summations.  It  is  interesting 
to  note  that  because  of  a  lack  of  symmetry  in  the  original 
form  of  the  bivariate  expansion,  the  causal  region  in  the 
regular  form  extends  outside  the  quarter  plane. 

As  was  the  case  with  the  Volterra  model,  it  will  be 
shown  that  in  regular  form,  a  minimum  mean  square  equation 
error  solution  can  be  obtained  for  the  nonlinear  ARMA  model 
coefficients  using  either  the  Levinson  algorithm  or  the 
lattice  methods.  This  will  provide  the  nonlinear  generali¬ 
zation  of  the  results  presented  in  Chapter  III  on  linear 
ARMA  modeling.  Before  developing  this  method  of  solution, 
however,  the  applicability  of  the  nonlinear  ARMA  model  to 
various  types  of  systems  and  its  memory  requirements  will 
be  considered. 

1 .  Identif lability  Conditions  and  Memory  Requirements 
In  the  previous  chapter  on  linear  ARMA  modeling  it 
was  stated  that  there  were  two  facets  to  the  question  of 
model  identif iability •,  input  signal  requirements  and  mea¬ 
surement  requirements.  The  question  of  measurement  re¬ 
quirements  was  not  discussed,  however,  since  it  was  assumed 
that  all  signals  (input  and  output)  were  observed.  In  the 
study  of  systems  comprised  of  interconnected  linear  and 
nonlinear  subsystems,  however,  various  internal  signals  exist 
and  the  effect  of  either  observing  or  not  observing  them  on 
the  modeling  process  must  be  explored.  To  do  so,  it  will 

t 

be  assumed  that  the  system  under  study  can  be  put  into  the 
form  of  Figure  4.8  fulfilling  the  following  equations. 
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(4.17a) 


x^(k)  =  £x  ^j(k)  ♦  C^u(k) 

Oc >  s  £2  £L(k)  +  C2u(k)  (4.17b) 

j^(k)  *  F  Cx^k)]  (4.17c) 

yL(k)  =  T  CxL(x)]  ( 4 . 17d) 

where 

xL(k)  =  CxL1(k)  ...  xLp(k)]T  (4.18a) 

a  vector  of  inputs  to  P  linear  subsystems; 

xN(k)  s  CxN1(k)  ...  xNQ(k)3T  (4.18b) 

a  vector  of  inputs  to  Q  nonlinear  memory less  subsystems ; 

^(k)  =  (4.18c) 

a  vector  of  outputs  from  P  linear  subsystems; 

^(k)  =  tyN1<k)  •••  yNQ(k)]  (4.18d) 

a  vector  of  outputs  from  Q  nonlinear  memoryless 
subsystems . 
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The  z  -  transforms  of  these  signals  are  also  defined  as 
XL(z),  XN<z),  Yl(z),  Y^(z).  £^,  £2 «  and  C2  are  matrices 

whose  elements  are  either  0,  -1  or  +1  indicating  the 
interconnections  of  the  various  subsystems.  T[.]  and  F[ . ] 
are  diagonal  matrices  specifing  the  linear  and  nonlinear 
subsystems  as  follows . 

Yl(z)  =  T(z)  XL<z>  (4.19a) 


where 


T(z)  = 


[Ti j ( z) ] 


A^(z) 

l-B.(z) 


1=] 


(4.19b) 


and 


. (z)  =  ^  a.(n)  z“n 

1  n=0  1 


(4.19c) 


N 

B . ( z )  =  X  b.(n)  z“n 
1  n=l  1 


( 4 . 19d) 


^(k)  =  F[x^(k)  ]  = 


F1C.3 


f2e.j 


FqC.1 


J 


(k) 

? 

xN:(k) 

X^g(k) 


FiCxN^(k) ] 


(4.20) 
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This  overall  system  given  by  Figure  4.8  is  adequate  to  re¬ 
present  a  broad  class  of  systems  comprised  of  intercon¬ 
nections  of  linear  and  nonlinear  subsystems  including 
cascades,  parallel  connections  and  feedback  systems. 

Equations  (4.19)  and  (4.20)  assume  all  the  subsystems 
are  single  input  single  output  noninteracting  systems.  If 
desired,  the  collection  of  linear  subsystems  given  by 
T ( 2 )  can  readily  be  put  into  the  general  multichannel  ARMA 
form  of  Section  III.F  to  allow  each  output  to  be  a  function 
of  past  values  of  all  outputs ,  and  past  and  present  values 
of  all  inputs. 

Alternate  representations  of  these  linear  and  non¬ 
linear  subsystems  are  also  useful.  Equations  (4.19)  are 
equivalent  to 

Yl(z>  =  CAi(z)]XL(z)  +  [Bi(z)]YL(z)  (4.21a) 

or  in  the  time  domain 

£L(k)  =  Ca.(k)]*xL(k)  +  Cb^k)  3*£L(k)  (4.21b) 

Here  *  represents  convolution  and  is  carried  out  in  the 
same  sense  as  matrix  multiplication  and  the  matrices. 

[a.(k)3,  Cb.(k)],  [A. (z)]  and  CB.(z)]  are  diagonal  matrices 
whose  i-th  entries  are  the  time  domain  functions  a^Ck)  or 
b^(k)  or  the  polynomials  A^(z)  or  B^(z).  The  time  domain 
functions  a^(k)  and  b^(k)  are  the  inverse  transforms  of  the 
polynomials  A. (z)  and  B.(z).  This  can  also  be  represented 

Ji  X 


**  p 


in  nonrecursive  form  as 


(k)  =  [h. (k) ]*x.  (x) 

X  -“*Li 


(4.22a) 


where  Ch^(k)]  is  a  diagonal  matrix  of  impulse  responses 
defined  by 


h. (k)  =  2  h. (n)  5 (k-n) 

1  n=0  1 


(4.22b) 


The  nonlinear  systems  can  also  be  represented  in  terms  of 
inverse  functions  assuming  they  exist  over  the  necessary 
ranges  of  the  variables  so  that 


3^00  =  r-H^Ocn  = 


0 

• 

1 - 

•  /> 

w 

r~ 1  • 
2 
i  * 

• 

0  *FqXC*  3 

• 

%5Ck)_ 

F^Cy^Ck)]' 


FQ  ^-Ffjq^k) ] 


(4.23) 


So  that  equations  (4.17)  are  iterative  it  is 
necessary  that  no  delay  free  loops  exist  in  the  system  of 
Figure  4.8.  A  necessary  j? tlrj ftftX  ftCttdXtXaa xu 
absence  of  delay  free  loops  is  developed  in  Appendix  H  and 
requires  that  the  terms  of  the  determinant  of  the  matrix 
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(4.24) 


a  =  Lim  £2  T(z) 

2 -►OB 


and  all  its  principal  minors  must  be  zero. 

The  various  signal  vectors  and  equations  (4.17)  can 
now  be  partitioned  as 


*{.(  k>“ 

«• 

Ila 

h* 

xA(w" 

£ia‘ 

x"(k) 

he 

lid 

— 

—  — 

£lb 

-2  b 

zi(k>’ 

+ 

=2a 

xJOO 

—2c 

-2d 

— 

^(k) 

£2b 

«• 

Fb[.T 

-*N0cr 

a(k)_ 

Ict-1 

x^(k) 

v-1  £bc-r 

lSL(kr 

*L<k> 

IcC.J  T^.] 

x£(k) 

u(k) 


u(k) 


(4.25a) 


(4.25b) 


(4.25c) 


( 4 . 25d) 


Equation  (4.25c)  can  also  be  written  in  terms  of  inverse 
functions  as  in  equation  (4.23),  and  (4.25d)  can  be  written 
using  either  recursive  or  nonrecursive  representations  of 
the  linear  systems  as  was  done  in  (4,21)  and  (4,22).  The 
single  primed  signal  veotors  in  equations  (4.25)  represent 
those  signals  which  are  observed  and  the  double  primed  signals 
are  those  which  are  not.  It  is  assumed  that  all  input  signals 
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in  u(k)  are  observed.  It  is  possible  to  rewrite  equations 
(4,25)  as  a  single  composite  matrix  equation  as  done  in 
(4.26a).  As  written,  equation  (4.2 6a>  is  an  infinite  memory  or 
model  order  representation  because  of  th  2  presence  of  the 
h(k)*  operators  but  a  finite  memory  version  can  be  written 
by  expressing  rows  4  and  8  as 

£t'  (k>  =  aa(k)*x^(k)  +  ba(k)*^(k)  +  a.b(k)*x£(k) 

+  bb(k)*y£(k)  (4.23b) 

y"(k)  =  a^k^x^k)  +  b,(k)^(k)  +  ad(k)*x£(k) 

+  b£(k)*jr£(k)  (4.26c) 

The  third  and  seventh  rows  can  also  be  written  in  terms  of 
inverse  functions  if  desired  as 


XjJCk)  =  F^t^Ck)]  *  (4.26d) 

xj(k)  =  F^CyJOc)]  +  £f[jrj;<k>]  (t.26«) 


Now  the  problem  of  writing  a  system  of  iterative  equations 
for  the  observed  signals  in  terms  of  only  observed  signals 
consists  of  rewriting  equation  (4.26e)  so  that  the  upper 


j  V  * 


partition  is  a  null  matrix.  If  -his  can  be 


do:-,«  „  the  general  frm  of  the  nonlinear  A?.V.A  model  car.  be 


used  to  identify  the  composite  effects  of  the  operators 
appearing  in  the  upper  left  5  by  5  partition.  In  some  cases 
this  will  only  be  possible  of  the  infinite  memory  version  of 
the  model  (with  h(k)*  operators)  is  used  and  in  other  cases 
a  finite  memory  model  will  suffice. 

The  process  of  determining  whether  infinite  or  finite 
memory  nonlinear  ARMA  models  can  be  used  to  identify  a  given 
system  is  illustrated  for  two  examples  in  Appendix  I .  First 
a  system  consisting  of  a  cascade  of  linear  and  nonlinear 
systems  is  considered.  Then  a  model  of  the  tracking  behavior 
of  a  phase  locked  loop  is  put  in  the  form  of  a  nonlinear  ARMA 
representation . 

2 .  Lattice  Filter  Methods  for  the  Nonlinear  ARMA  Model 
As  was  the  case  with  Volterra  modeling,  lattice 
filter  methods  can  be  applied  to  the  nonlinear  ARMA  modeling 
problem  if  the  regular  form  of  the  kernels  is  used.  A 
family  of  signals  is  defined  as  nonlinear  combinations  of 
delayed  values  of  y(k)  and  u(k)  as  follows. 


u,  .  (k)  *  u(k)  u(k-h,, )u(k-h,-h- ) . .  ,u(k-h-- .  . . -h  ) 

ru.  •  . h.  2  z  3  2  m 

z  m 

(4.27a) 

y.  k  Cfc)  2  y(k-l)y(k-l-h,)y(k-l-h,-h,) 

n2  * ' *  nm  z  2  j 

. . .y (k-l-h-- . . . -h) 

l  Rl 


ill 


(4.27b) 


u,  ,  y.  ,  (k)  =  u(k)  u(k-h~) . . .u(k-h0-. . .-h  ) 

h2---hP  Vi-V?  2  2  p 


yCk-l-hj-.  •  •-hp+^) 


. .  .yCk-1-^-. .  .-hp+^)  (4.27c) 


or  alternately 


h2‘**hq  hq+l*'*^q+p  s  y (k-l)y ( k-*l-h2 ) . .  ,y (k-l-h2- •  • . -h  ) 


u(k-h2** •  * . ~hq+2_)  •  •  •  u(k-*h2*,*»  *  •  '-*^q4p ) 


( 4 . 27d) 


With  finite  summations,  the  regular  form  of  equation  (4.14) 
becomes 


Vp 

2 

h  =0 
P 


V2 

2 

h2  =  0 


N 


,1 


hL=0 


a  (h, ...h  )  u,  .  (k-h.  ) 

r  1  p  h2 . .  . hp  l 


(4.28a) 


Equation  (4,  IS)  becomes 


Mq,q 
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hq-° 


M. 


,2 


h2*o 
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b_(h, . ..h_) 
Is 


r(h1...h  )  yh  h  - 1 

h,  *0  2  °- 


(k-h^ 

U.29b) 


•  t  • 


and  equations  (4.16)  become 


Lp+q,p+q 

2 

=0 


LP+q,2 

2 


h2  =  0 


1 


=  0 


or 


(k-hx) 


(4.28c) 


Lq+p,q+p 


Lq+p » 2 

2 

h2  =  0. 


Lq+p,l 

2  Cr2(hl,*,h q+p5  yh ,...h 

h  ”0  i  q 

1 


(4 . 28d) 


These  terms  can  be  viewed  as  the  summation  of  the  outputs 
of  a  large  number  of  linear  filters  whose  inputs  are  the 
signals  defined  in  equations  (4.27).  In  the  context  of 
multichannel  filtering,  each  cf  these  filters  can  be  con¬ 
sidered  as  a  single  channel  and  each  of  the  input  signals 
can  be  associated  with  one  of  the  channels.  Since  the 
lattice  models  use  the  same  amount  of  memory  in  each  channel, 
the  upper  limits  on  all  summations  over  will  be  made  the 
same  (H^  i  s  ^  =  N^).  The  upper  limits  on  the 

suanat  lone  over  the  other  indicies  determine  the  number  of 
ihanne 1 s  requ ir ed . 

is : 


ror  a  quadratic  nonlinear  system,  the  present  value 
of  the  system  output  is  estimated  as 


%  a  (h, )u(k-h2)  t  X  2  ar(h1,h2)uh2(k-h1) 

IT  A  a  n 


y(k)  = 


hr-° 


h2=0  h1=0 


+  ^  b  (heiyCk-^)  +  2  2  br(hl’h2)yh2(k“hl) 

hx-l  r  V°  V° 


u22 

+  2 

V-o 


Ni 

X  (hl’V  U  yh,(k"hl' 


(4.29) 


v° 


where  (k>  =  uOOutk-hj)  ,  y^CM  -  yU-l>y<k-l-h2> 

a„d  u  y.2 (k)  =  U(k)y(k-l-h2).  Signal  and  coefficient  vectors 
oan  he  defined  for  the  various  quantities  in  equation  (4.29) 
following  the  conventions  established  earlier,  for  exa  p 


£(k) 


=  [y(k-l)  yCk-N^] 


(k-N, )3 


,+  (k)  =  Cyh  Ck)  yh, 


b  =  Cbr(l)  • • •  br(Ni)] 


b*  *  Cb  < 0 ,k  )  ...  VW] 

— n 1  L 


(4.30a) 


(4.30b) 


(u . 30c) 


( 4 . 30d) 
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Embedding  all  these  vectors  into  single  data  and  coefficient 
vectors,  equation  (4.29)  becomes 


y(k)  s  _X1(k)T  dx 


(4.31a) 


where 


X.x(k)  =  [y.(k)T  1  u+(k)T  Zo(k)T  | 


+  T 

•  Zm  (1c) 


+  ,T  !  '  +  /^nTI 

uQ(k)  ,  ...  ,  ^  (k)  1 

l  t  22 


+  T  •  I  +  T  ' 

u£Q(k)  |  . . .  {  uyL  (k)  ] 


(4.31b) 


b+T  ' 

k0  ! 


i  + 

I  — N, 


— 0  t 


'  +  T  1 
•  I  £l  ] 

,  L22 


(4.31c) 


The  minimum  mean  square  equation  error  solution  for  the 
model  coefficient  vector  d^  is  given  by 


e {  X^(k)  ll  s  e(Xi(k)  y(k) } 


(4.32) 


where  the  equation  error  is  defined  as 


e(k)  s  y(k)  -  y(k) 


13*» 


While  equation  (4.32)  is  similar  in  form  to  equations 
(A. 7a)  and  (A. 26a)  for  AR  and  MA  models,  it  lacks  the  nec¬ 
essary  form  for  the  application  of  the  Levinson  algorithm 
because  of  the  fact  that  some  component  data  and  coefficient 
vectors  are  N^-vectors  while  others  are  (N^+l) -vectors . 

This  is  the  same  problem  that  arose  in  linear  ARMA  modeling 
and  can  be  handled  here  in  a  similar  manner.  Defining 


i*  Cl  -ar(0)  -  br(0,0)  ...  -br(0,M22) 

-  ar(0 ,0)  . . .  -a^CO ,N22) 

T 

“  crl(0,0)  ***  ~crl(0’L22):l 

(4.33a) 

and 

x(k)  =  Cy(k)  u(k)  yn(k)  •••  yM 
-  u  n22 

un (k)  ...  uM  (k) 

u  ^22 

T 

uyQ(k)  ...  uyL  (k)'J 

2  2 

(4.33b) 

and  assuming  that  the  coefficients  in  £  can  be  estimated  in 
some  other  manner,  the  MMSE  solution  for  the  remaining 
model  coefficients  becomes 


1SS 


(4.34) 


e(]£  (SO  1  =  e  { X(k)  x(k)^}  £ 

Here  X(R)  and  d  are  defined  as  _X^(k)  and  d^  with  all 
superscripts M+M  removed  from  their  component  vectors  (indi¬ 
cating  they  are  all  indexed  from  1  to  and  are  all  N^- 
vectors).  Note  that  just  as  in  the  linear  ARMA  case,  the 
coefficients  in  tj;  essentially  correspond  to  gains  on  their 
respective  channels.  Comparing  equation  (4.34)  with  (A. 7a) 
for  a  multichannel  autoregression  it  is  clear  that  d  can  be 
obtained  from  the  multichannel  AR  solution  by 

d  :  Di  *  (4.35) 

where  the  signals  in  x(k)  comprise  the  channels  in  the  auto¬ 
regression.  Therefore,  either  the  Levinson  algorithm  or  the 
lattice  methods  can  be  used  to  solve  for  _D  and  from  it  and 
a  knowledge  of  the  nonlinear  ARMA  model  coefficients  can 
be  obtained. 

By  analogy  with  equation  (3,27)  it  also  follows  that 
the  nonlinear  ARMA  equation  error  is  related  to  the  AR  pre¬ 
diction  error  vector  by 

e(k)  =  £T  e(k)  (4.36a) 

so  that 

tle(k)2}  =  *T  P  <p  (4.36b) 
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where  P  is  the  AR  prediction  error  covariance  matrix.  The 
coefficients  in  ^  can  therefore  be  set  to  minimize  the  mean 
square  value  of  the  nonlinear  ARMA  equation  error  in  (4,36b) 
resulting  in  a  solution  given  by 


(4.37a) 


where  N  is  the  total  number  of  channels  in  the  model 


N  *  1  +  1  +  (M22+l)  +  <N22+1)  +  CL22+1> 
and  the  p..  are  the  elements  of'  the  prediction  error  co- 

■*  J 

variance  matrix.  It  is  readily  apparent  that  the  linear 
ARMA  model  and  its  solution  via  the  Levinson  algorithm  or 
lattice  methods  are  a  special  case  of  this  formulation  of 
the  nonlinear  ARMA  model  just  as  one  would  expect. 
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V.  APPLICATIONS,  CONCLUSIONS,  AND  OPEN  QUESTIONS 


In  the  previous  four  chapters ,  existing  methods  for  AR 
and  MA  modeling  were  reviewed  and  from  them  new  methods  for 
linear  and  nonlinear  ARMA  modeling  were  developed.  Here, 
two  applications  for  these  new  methods  in  reduced  order 
modeling  and  modeling  for  fault  detection  and  diagnosis 
are  examined  briefly.  Then  the  results  of  this  research 
are  summarized,  conclusions  drawn  and  significant  open 
questions  for  the  continuation  and  extension  of  this  work 
are  listed. 

A.  APPLICATIONS 

1.  Reduced  Order  Modeling 

Oftentimes,  complex  physical  systems,  both  linear 
and  nonlinear,  can  be  approximated  quite  closely  using 
simple  models.  The  lattice  solution  methods  developed  here 
provide  a  very  natural  and  efficient  means  of  determining 
reduced  order  models  for  complex,  high  order  systems  especially 
in  the  case  of  linear  systems.  In  Chapter  III  it  was  argued 
that  for  linear  systems  it  is  reasonable  to  build  up  ARMA 
models  by  simultaneously  incrementing  the  order  of  the  numera¬ 
tor  and  denominator  polynomials  as  the  lattice  method  does. 

When  this  method  is  used  to  build  up  a  given  order  model, 
all  lower  order  models  and  their  mean  square  values  of  equation 
error  are  readily  obtained  as  well  (the  only  additional 
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calculations  needed  are  for  the  MSE  and  the  gain  term  or 
gain  matrix  in  the  multichannel  case)  making  it  easy  to 
compare  the  various  models  and  decide  if  reduced  order  models 
provide  sufficient  accuracy. 

Consider  for  example  the  seventh  order  system  whose 
characteristics  are  listed  in  Table  5,1.  The  magnitude 
spectra  of  second,  third,  sixth  and  seventh  order  lattice 
models  obtained  using  batch  processing  with  4-000  point 
averages  are  compared  to  that  of  the  system  in  Figure  5.1. 

It  is  apparent  that  a  second  order  model  is  unable  to 
approximate  the  system  well,  however  a  third  order  model  does 
provide  a  good  approximation.  Furthermore,  increasing  the 
model  order  to  four,  five  and  six  fails  to  significantly 
improve  its  -performance  as  evidenced  by  the  sixth  order 
plot  in  Figure  5.1c.  The  model  accuracy  is  not  significantly 
increased  until  seventh  order  (which  corresponds  to  the  order 
of  the  actual  system)  when  a  very  good  fit  is  achieved.  This 
is  further  illustrated  by  the  plot  of  the  normalized  mean 
square  equation  error  as  a  function  of  model  order  shown  in 
Figure  5,2.  The  cost  drops  rapidly  going  from  second  to 
third  order  but  then  fails  to  decrease  significantly  until 
seventh  order  is  reached.  The  roots  of  the  system  and  the 
various  order  models  are  also  plotted  in  Figure  5.3. 

The  benefits  of  the  lattice  method  for  reduced  order 
nonlinear  ARMA  modeling  are  not  quite  so  pronounced  however, 
since  adding  extra  stages  to  the  lattice  corresponds  to 
allowing  the  kernels  to  grow  in  only  one  of  their  multiple 
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TA8LE  5.1 


SYSTEP  E 

TRANSFER  FUNCTION  COEFFICIENTS 


NUMERATOR 


A  ( C  )* 

l.COOOO 

Alii* 

-2.16510 

A  (2)* 

1.5625C 

A  (21* 

o.o 

A  (4)* 

0.0 

A  (51* 

0.0 

A  (6)* 

0.0 

A  (7)  * 

0.0 

ROOT 


ZEROS 

RE 

1M 

1.CS25C 

0.6250C 

I.C8250 

-0.62500 

C.C 

0.0 

c.c 

o 

. 

o 

c.o 

0.0 

C.C 

0.0 

c.c 

0.0 

OSNCMIMTCR 

BID*  2.18950 
8(2)*  -2.21260 
6(3)*  1.51740 

6(4)*  -C.85350 
B ( 5 ) *  C. 45144 
8(61*  -C.23462 
8(7)*  0 .09331 

locations 

pciss 

RE  ** 

Q.9C000  0.0 

0.65282  C.4000C 

0.69282  -0.4000C 

0.23941  0.65778 

0.22941  -0.65778 

-0.28750  0  49796 

-0.28750  -0.49796 


ir 


n/2 

I  H  (•)  I  0B  VS  ♦  —  SYSTEM 

—  2  NO  ORDER  LATTICE  MODEL 
4000  POINT  AVERAGES 


0  »/2 
■  H  (♦)  I  0B  VS  ♦  —  SYSTEM 

—  3  RD  OROER  LATTICE  MOOEL 
4000  POINT  AVERAGES 


Figure  5.1 


dimensions.  (The  linear  "kernel"  had  only  one  dimension.) 

Here  some  a  priori  information  would  be  useful  in  determining 
the  values  to  prespecify  for  the  boundaries  of  the  kernels 
in  the  other  dimensions.  Still  however,  when  compared  to  a 
brute  force  matrix  inversion  approach  where  all  kernel 
boundaries  must  be  prespecified,  the  lattice  method  provides 
a  viable  alternative  in  obtaining  reduced  order  models 
especially  for  low  degree  systems. 

2 .  Modeling  For  Fault  Detection 

The  problem  of  fault  detection  and  possible  diagnosis 
can  be  formulated  as  follows. 

a.  Obtain  a  parameterization  that  describes  the 
current  functioning  of  the  system  under  test. 

b.  From  this  parameterization,  determine  if  the 
system  is  functioning  normally  or  if  a  fault 

has  occurred  by  comparison  with  a  fault  dictionary. 
It  is  the  first  part  of  this  problem  that  has  been  addressed 
in  this  work.  The  parameterization  can  be  as  simple  as 
sampled  measurements  of  the  response  to  specific  inputs 
however,  the  large  volume  of  data  that  would  generally  be 
involved  in  such  an  approach  would  greatly  complicate  the 
second  part  of  the  problem.  A  more  efficient  approach  in 
terms  of  utilization  of  parameters  is  to  model  the  system 
and  use  the  model  parameters  as  a  description  of  its 
current  functioning. 

For  linear  systems,  ARMA  models  provide  a  very 
general  framework  with  a  number  of  possible  parameter  sets. 


Three  candidates  are  polynomial  coefficients,  root  locations 

and  lattice  reflection  coefficients  with  the  latter  offering 

many  advantages .  In  addition  tc  the  advantages  demonstrated 

by  the  experimental  results  of  Chapter  III,  reflection 

coefficients  provide  a  very  efiective  and  methodical  way  to 

build  up  knowledge  of  a  system's  characteristics.  As  model 

order  is  increased  to  more  accurately  represent  a  system, 

reflection  coefficients  already  determined  don't  change 

making  them  ideal  candidates  for  use  in  a  dictionary  lookup 

scheme  (a  characteristic  not  shared  by  the  other  parame- 

terizations ) .  This  is  made  more  important  since  reduced 

order  models  could  be  adequate  to  detect  and  perhaps  diagnose 

some  faults,  especially  catastrophic  ones.  While  moj'e 

parameters  are  required  when  reflection  coefficients  are 

used,  these  same  coefficients  also  provide  all  reduced  order 

models.  For  a  single  channel  ARMA  example,  8N  reflection 

coefficients  and  N  gains  provide  all  models  from  order  1  to 
2 

N  while  N  +2N  parameters  would  be  required  using  either 
polynomial  coefficients  or  roots  to  provide  the  same  infor¬ 
mation.  (6N  reflection  coefficients  are  needed  if  the  input 
is  white  noise  since  k^2=k22=0.) 

A  similar  argument  could  be  made  for  the  use  of 
lattice  reflection  coefficients  with  the  nonlinear  ARMA 
model  for  fault  detection  and  diagnosis  of  nonlinear  systems. 
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B.  CONCLUSIONS  AND  OPEN  QUESTIONS 

The  purpose  of  this  research  was  to  extend  existing 
theories  and  methods  in  the  modeling  of  linear  and  nonlinear 
systems  to  broader,  more  general  types  of  models.  After  a 
discussion  of  available  results  in  AR  and  HA  modeling  of 
linear  systems  with  particular  emphasis  on  the  Levinson 
algorithm  and  lattice  filter  methods,  model  transition 
formulas  were  developed  to  relate  the  more  general  ARMA 
model  for  linear  systems  to  the  AR  and  HA  models.  It  was 
shown  that  with  suitable  assumptions,  the  ARMA  model  solu¬ 
tion  could  also  be  obtained  recursively  in  order  using 
either  a  modified  Levinson  algorithm  or  lattice  filter 
methods.  These  results  were  developed  extensively  in  both 
theory  and  practice  for  single  channel  linear  ARMA  modeling 
with  experimental  verification  of  both  the  batch  processing 
and  adaptive  lattice  methods  presented.  Portions  of  these 
results  have  already  been  published.  [Refs.  65  and  66] 

The  theory  was  also  developed  to  generalize  these  results 
to  the  multichannel  ARMA  case. 

Based  on  the  simulation  results  it  was  concluded  that 
the  lattice  methods  offer  the  following  advantages  over  a 
conventional  brute  force  matrix  inversion  approach  to  ARMA 
modeling  using  windowed  correlation  estimates. 

1.  For  short  runs  of  data  the  batch  lattice  methods 
provide  much  more  accurate  results  than  the  brute 
force  method. 
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2 .  The  batch  lattice  method  performs  much  better  than 
the  brute  force  method  when  the  system  is  over¬ 
modeled  , 

3.  The  MSE  as  a  function  of  model  order  is  well  behaved 
for  the  lattice  method. 

4.  The  adaptive  lattice  method  has  difficulty  tracking 
zero  valued  overmodeled  parameters. 

The  cost  of  these  advantages  is  the  extra  computational 
burden  of  passing  the  data  through  the  lattice  filter  during 
the  modeling  process. 

In  the  discussion  of  nonlinear  system  models  the  Volterra 
model  was  considered  as  a  nonlinear  extension  of  MA  modeling 
and  it  was  shown  that  lattice  methods  could  be  used  to  obtain 
the  model  solution  if  the  problem  was  recast ,  using  the 
regular  form  of  the  Volterra  kernels.  Then  the  new  nonlinear 
ARM/*  model  was  considered  and  it  was  shown  that  this  repre¬ 
sentation  in  some  cases  solves  the  problem  of  requiring  a 
very  large  number  of  model  parameters  encountered  in  Volterra 
modeling.  Then  lattice  methods  were  developed  for  the  non¬ 
linear  ARMA  problem  and  it  was  shown  that  the  linear  ARMA 
techniques  presented  earlier  are  a  speical  case  of  the  non¬ 
linear  ARMA  methods.  For  both  types  of  nonlinear  models, 
the  recursive  in  order  nature  of  the  lattice  methods  was 
shown  to  allow  the  various  model  kernels  to  grow  in  one 
dimension  while  holding  their  boundaries  fixed  at  pre¬ 
specified  values  in  the  other  dimensions.  The  use  of  the 
model  nonlinear  ARMA  was  also  illustrated  with  two  examples 
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and  a  nonlinear  ARMA  model  was  proposed  for  the  tracking 
behavior  of  a  phase  locked  loop. 

Several  significant  questions  remain  for  the  continuation 
and  extension  of  this  work  and  are  listed  here, 

1.  Stability  of  the  linear  and  nonlinear  ARMA  models 
must  be  considered.  In  the  linear  problem,  stability 
is  dependent  on  the  roots  of  the  demoninator  poly¬ 
nomial  of  the  synthesis  model.  The  methods 
developed  do  not  guarantee  stability  of  the  resulting 
model.  (This  was  not  found  to  be  a  problem  in 
practice,  however ,  unless  extremely  short  runs  of 
data  in  the  range  of  30  to  50  samples  were  used. 

Even  then,  model  instability  was  not  a  frequent 
occurrence.)  Stability  for  the  nonlinear  ARMA  model 
remains  to  be  clearly  defined. 

2.  Input  signal  requirements  in  the  nonlinear  ARMA 
modeling  process  need  to  be  investigated.  In  linear 
ARMA  modeling  the  power  spectrum  of  the  input  signal 
was  found  to  play  an  important  role.  No  requirements 
emerged  however,  on  the  probability  density  function 
(pdf)  of  the  input.  In  nonlinear  systems  where  the 
behavior  is  inherently  level  dependent ,  it  is  in¬ 
tuitively  appealing  to  use  an  input  signal  with  a 
flat  power  spectrum  across  the  frequency  range  of 
interest  and  whose  amplitude  is  uniformly  distributed 
over  the  range  of  interest. 
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3 .  The  inability  of  the  adaptive  lattice  method  to 
track  zero  valued  overmodeled  parameters  is  an 
interesting  problem  warranting  further  consideration. 
If  some  means  of  detecting  the  degeneration  of  the 
cost  surface  towards  an  infinite  trough  can  be  found, 
the  problem  could  be  remedied  by  simply  reseting  the 
appropriate  parameters  to  zero. 

4.  Experimental  experience  needs  to  be  gained  with  the 
nonlinear  ARMA  model  itself  and  the  lattice  methods 
developed  for  it. 

5.  The  characteritics  of  the  lattice  solution  methods 
need  to  be  further  quantified  to  gain  a  comprehensive 
understanding  of  how  and  why  it  performs  as  it  does. 
Also ,  further  comparisons  should  be  made  between 

the  lattice  methods  and  conventional  methods .  Some 
comparisons  were  made  here  for  batch  processing 
methods  but  only  a  rectangular  window  was  used  on 
the  data.  Comparisons  should  be  made  using  other 
types  of  window  functions  in  the  brute  force  method 
and  the  adaptive  lattice  method  should  also  be  com* 
pared  with  a  conventional  LMS  adaptive  algorithm 
applied  to  the  equation  error  model  in  which  all  of 
the  a(i)  and  b(i)  coefficients  are  adapted  simul¬ 
taneously, 

6.  In  the  adaptive  lattice  method ?  scaling  of  the  lattice 
input  signals  needs  to  be  investigated.  It  was 
noted  that  the  ratio  of  largest  to  smallest  eigenvalue 
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of  the  input  autocorrelation  matrix  was  related  to 
the  speed  of  convergence  of  the  adaptive  algorithm. 
For  the  first  lattice  stage  this  matrix  .is 

y2(k)  y(k)  uCk) 

PC0)  =  e{  } 

u(k)  yCk)  u2(k) 

If  the  system  has  a  high  gain  such  that  the  mean 
square  value  of  the  output  y(k)  is  very  much  greater 
than  that  of  the  input  u(k) ,  convergence  will  be 
slow.  This  could  be  remedied  by  implementing  an 
adaptive  scaling  scheme  at  the  lattice  input  (perhaps 
similar  to  the  first  order  low  pass  filter  estimates 
used  for  adaptive  step  size  normalization) . 
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Alternate  Multichannel  Model  Forms 


Multichannel  generalization  of  the  MA  and  AR  models  were 
discussed  in  Chapter  II  along  with  their  solution  via  the 
Levinson  algorithm.  Here  the  multichannel  models  and  the 
Levinson  algorithm  are  developed  in  an  alternate  form  more 
compatible  with  other  linear  and  nonlinear  modeling  problems 
to  be  explored  later. 

Consider  first  an  N-th  order,  Qq -channel,  AR  model  where 
the  current  value  of  the  signal  vector  x(k) 


T  ' 

x(k)  =  [x1(k) 

...  x  (k)] 

(A. la) 

T 

X(z)  =  [X^Cz) 

...  Xn  (y) ] 

^0 

(A. lb) 

is  to  be  predicted  from  a  weighted  combination  of  N  past 
values  of  each  of  the  component  signals.  For  each  signal 
this  estimate  can  be  written  as 

Q0  n 

V-l  dn.^(n)  x^(k-n)  (A, 2a) 


Xj(k)  = 


1=1  n-1 


or 


X .  (k)  =  d . .  ( z )  X.  ( z) 

3  1^1  1 


where 


N 


13  13 


-n 


(A2.b) 


(A. 2c) 
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Define  an  N-vec.ton  for  each  of  the  Qg  channels  to  contain 
their  required  time  histories  as 


x. (k)  s  Cx.(k-l)  ...  x.(k-N)] 

•1*  JL 


(A. 3a) 


for  1  <  i  Qg  and  embed  all  of  these  vectors  into  a  single 
NQg-data  vector  given  by 


X‘<k)  =  Cx,(k)T  ...  Xq  Ck)T] 


(A. 3b) 


Define  a  NQq  x  Qq  matris  of  weights  as 


D  =  dn  ...  d1Q 


(A. 4a) 


'V 

1^0 


where  the  N-vectors  are  given  by 


d..  3  Cd..(l)  ...  d . . ( N ) ] ' 

—XT  1]  13 


(A. 4b) 


and  contain  the  coefficients  of  the  polynomials  d^j(z). 
These  polynomials  can  be  combined  in  a  matrix  polynomial 
defined  by 


du(z) 


Dcz)  = 


H(z) 


(A. 4c) 


vu> 


dWz) 
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With  these  definitions »  the  N-th  order  prediction  of  x(k) 
and  its  associated  prediction  error  vector  becomes 


x(k)T  =  X.^T  .0 

rp  rn  /,  rp 

e(k) 1  =  x'k)  -  x(k) 


(A. 5a) 


(A. 5b) 


or  in  the  transform  domain 


E(z)T  =  X(z)x  Cl  -  D(z)] 


(A. 5c) 


a*v- 


Comparison  of  equation  (A.  5c)  and  C2.44c)  show  that  this 
matrix  polynomial  differs  from  the  more  generally  used  form 
of  (2.44c)  by  a  transposition.  The  coefficient  matrix  D 
can  be  found  by  minimizing  the  trace  of  the  prediction 
error  covariance  matrix 


P  =  e  (e(k)  e(k)  } 

leading  to  a  system  of  linear  equations  given  by 

e{X(k)  X(k)T}  D  =  e(XCk)  xOOT} 

or 


(A. 6) 


(A. 7a) 


R,  ... 

~xlxl 

*x  x 
xixq0 

-XX 

xlxl 

E^o_ 

H x  x  *  *  * 

_  Q0  1 

r* 

%  \ 

D  -- 

% 

• 

• 

-X  X 

__  Qq  1 

• 

• 

-X  X 

V’oJ 

(A. 7b) 
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Adopting  a  shorthand  notation,  this  becomes 


R  D  = 


r 


(A.?c) 


As  in  the  case  of  a  single  channel  autoregression,  the 
multichannel  generalization  of  the  Levinson  algorithm  also 
requires  that  the  backward  prediction  problem  of  estimating 
x(k-N)  backward  in  time  from  the  values  of  x(k-N+l)  through 
x(k) ,  be  solved  si 'ultaneously.  Using  an  overscore  to 
indicate  quantities  associated  with  the  backward  problem  it 
follows  that 


_  T  _  t 

X  (k)  D  =  x(k) 

T  T  A  T 

e(k)  =  x(k-N)  -  x^kO 

or 

IT  _ 

f(z)  =  X(  z)  z”H[I  -  D(z)3 

where 

X(k)  ~  [x, (k)  ...  x.  Ck)T] 

-1  “WQ 

T 

X.  (k)  =  [x. (k-N+1)  ...  x,(k)] 

and 

D  In  • , .  d1Q^ 

•  • 

n  • 

•  • 

^o1 

(A. 8a) 

(A. 8b) 

(A. 8c) 

(A.8d) 

(A.8e> 

<A.8f) 
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d.  .  =  [d.  ,  (1)  3\.(N)] 


(A.8g) 


with  Dcz)  = 


^11 5 


V*’ 


V<2>  *«o<»o 


(A, 8h) 


and 


a..(2> 


N 

-i 

n=l  ^ 


=  1j  2T.  .(n)  zn 


(A. 8i) 


Setting  the  coefficients  of  this  backward  predictor  to  mini¬ 
mize  the  trace  of  the  backward  prediction  error  covariance 
matrix 


F  s  e{e(k)  e(k)T}  (A. 9) 

leads  to  a  MMSE  solution  given  by 

e{X(k)  X(k)T>  fi  s  E^X(k)  x(k-N )T }  (A. 10a) 

—  —  T  T 

and  since  e  lx .  (k)  x.,,  }  =  R 

—1  —jKKJ  "~Xi  3 

and  elx. <k)  x. (k-N)}  =  r 

-1  -3  “*xjxj, 

this  can  be  written  as 


ill 


Adopting  a  shorthand  notation,  this  becomes 


(A. 11) 


At  this  point  it  is  important  to  take  note  of  a  subtle 
difference  between  the  single  and  multichannel  problems ‘that 
has  arisen.  While  the  single  channel  equivalent  of  equation 
(A. 10b)  for  the  backward  predictor  was  not  written  previously, 
it  is  obviously 


Rxx  l  s  Ix  x 

xlxl  xlxl 


and  since 


rt 

-*1*1 


-*1*1 


(A. 12a) 


(A. 12b) 


it  follows  that  the  single  channel  forward  and  backward 
prediction  problems  have  exactly  the  same  solution  (as  was 
found  to  be  the  case  in  Section  II. D.  when  the  AR  prediction 
error  lattice  was  developed) .  This  fact  is  responsible  for 


w* 


..^3 
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a  number  of  simplifications  in  the  single  channel  case: 

a. )  Only  a  single  reflection  cof ffj,cient  k/n+^  was 

needed  to  link  the  n-th  and  (n+l)-st  order  solu¬ 
tions  for  both  the  backward  and  forward  problems 
(see  equations  (2.25b)  and  (2.25c)). 

b. )  |B(z)j  =  |E(z)| 

c. )  e{e(k)2}  =  e{e(k)2} 

d. )  The  development  of  the  Burg  method  and  the  Itakaura- 

Saito  method  for  calculating  the  reflection 
coefficients  are  a  direct  consequence  of  c  above. 
Unfortunately  however,  none  of  these  simplifications  carry 
over  into  the  more  general  multichannel  AR  problem  because 
by  comparing  equations  (A. 7b)  and  (A. 10b)  it  is  evident 
that  in  general  D  *  D  . 

Proceeding  as  in  the  single  channel  case,  it  is  shown 
in  Appendix  B  that  because  of  the  structure  in  the  correlation 
matricies  JR  and  JR  ,  equations  (A. 7c)  and  (A. 11)  can  be 
re-written  as 

JR  JF  =  £_  (A. 13a) 

and 

R,  _F  *  P_  (A. 13b) 

where  JF,  JF_,  and  are  obtained  from  D,  JD  ,  _T_  and 
f  respectively  by  taking  their  component  vectors  and  turning 
them  upside  down  in  place;  i.e. 


with 


£l«o 


^n1  ^n<3, 


(A. 14a) 


where 


Cd . . <N)  . . .  d, .<1)3 

1 J  1  J 


(A. 14b) 


£.v  v  •  •  * 

xlxl 


£x  x 

1  Q0 


(A. 14c) 


Vxi 


S0xqJ 


^X.X.  =  CR, ...  (N)  ...  R  ..  (1)] 


i  3  xiXj 


xixj 


(A. 14d) 


As  will  soon  be  evident,  the  relationships  of  equations  (A. 13) 

form  the  cornerstone  for  the  Levinson  algorithm  and  are  made 

possible  because  the  blocks  comprising  the  _R  and  JR  matricies 

T 

are  themselves  Toeplitz  and  because  R„  v  =  Rv  „  . 

~xiXg  “xjxi 

Assume  that  the  (n+l)-st  order  solutions  can  be  related 


to  the  n-th  order  solutions  by 


"  (n) 
e  •  . 


k(n+D 
_ij  . 


(A. 15a) 
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-?(n+l) 

-ij 


a'?5 


«<?> 

-ID 

mm  m  mm 

^(n+1) 


(A, 15b) 


Embedding  the  vectors  and  -ij^  and  the  coef ^c^eivts 

kf1?*^  and  into  matrices  designated  and  C*n\ 

I]  1] 

^(n+1)  and  ^(n+1)  soiving  for  them  in  the  (n_l)-st  order 

problem  it  is  shown  in  Appendix  C  that 


.(n)  _  _  p(n)  K(n+1) 


(A. 16a) 


jj*(n)  _  __  p(n)  ^(n+1) 


(A. 16b) 


K(n+1)  =  [r  co)-  r (n)TD(n)]  [R„„(n+i)-/?(n)TD(n>3 


(A. 16c) 


K(n+1)  =  CR  v(0)-X(n)TD(n)]  X[R  (n+l)T-_£(n)TD(n)] 


(A. 16d) 


Also  the  inverted  terms  on  the  right  hand  side  of  equations 
(A. 16c)  and  (A.16d)  are  just  the  backward  and  forward  pre¬ 
diction  error  covariance  matrices  for  the  optimum  n-th 
order  models  so  that  (A. 16c)  and  (A.IBd)  become 


KCn+1)  =  PCn)  X[R  (n+.l)-/tf(n)rD<n)]  (A. 17a) 

K(n+D  =  P(n)  1[R  v(n+l)T-/<?Cn)TD(n)]  (A. 17b) 

'  A  X  '  "  "  ’  """ 


As  in  the  ca3e  of  the  single  channel  problem,  the  prediction 
error  covariance  matrices  obey  the  recursions  . 


p(n+l)  _  p(n)  [i-E^n+D^Cn+l)  -j 


(A. 18a) 


p’Cn+l)  _  p(n)  j-I-K(n+l)^(n+l)  ^ 


(A. 18b) 


completing  the  multichannel  generalization  of  the  Levinson 
algorithm  for  AR  models. 

Comparing  equations  (A. 16),  (A. 17)  and  (A. 18)  with  their 
single  channel  counterparts  (2.18)  and  (2.19)  it  is  clear 
that  the  multichannel  Levinson  algorithm  simply  represents 
a  matrix  algebra  generalization  of  the  single  channel  al¬ 
gorithm.  Once  again,  predictors  of  all  orders  0  _<  n  _<  N  are 
obtained  in  the  process  of  finding  the  N-th  order  predictor 
along  with  all  their  prediction  error  covariance  matrices 
and  the  overall  minimization  requiring  the  inversion  of 
a  QqN  x  QqN  matrix  is  replaced  by  a  sequence  of  N  minimiza¬ 
tions,  each  requireing  the  inversion  of  two  x  Qq  matrices 
(P(n)  and  F(n)). 
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Using  the  relationships  given  in  equations  (A. 15)  and 
(A. 16),  successive  orders  of  matrix  polynomials  can  also 
be  related  to  one  another  by 

cx-D<n+1,u>]  =  ti-D<n)(z)]-z-Vnci-D(n)(z)]K(ntl) 

(A. 19a) 

I-(n+1>Cl-D(n+1>(z)>z-1z-nCl-D(n)(z)]-tI-D(n)(z)]K(n+X) 

(A. 19b) 

T 

Premultiplying  both  sides  of  these  equations  by  X(z)  ,•  trans¬ 
posing,  and  transforming  into  the  time  domain  provides  rela¬ 
tionships  between  the  prediction  error  signals  at  each  stage. 

e<n+1)(k)  =  e(n)(k)  -  K(n+1)T  e(n)(k-l)  (A. 20a) 

e(n+1)(k)  =  e(n)(k-l)  -  KCn+1)  e(n)(k)  (A. 20b) 

Recognizing  that  for  a  zeroth  order  prediction,  the  forward 
and  backward  prediction  error  vectors  are  just  the  input 
vector  itself,  it  follows  that 

e(0)(k)  =  e(0)(k)  =  x(k)  (A. 20c) 


and  the  multichannel  equivalents  of  equations  (2.28)  have 
been  obtained. 


Next  consider  the  N-th  order  multichannel  HA  model  in 


which  the  current  value  of  a  Q^-vector  of  output  signals 
^(k)  is  to  be  predicted  from  the  present  value  and  N  past 
values  of  a  Q^-vector  of  input  signals  x(k) . 

9i  N 

y^(k)  =  ^ ^  23  d^j(n)  x^(k-n)  (A. 

i=l  n=0 

Qi 

°r  a  V  + 

Y .  (z)  =  dT ^ (2)  X.(z)  (A. 

i=l 


(A. 21a) 


(A. 21b) 


where 


T.(z)  =  23  d..(n)z~n 


(A. 21c) 


Using  a  superscript  ,}  +  "  to  indicate  the  fact  that  the  vectors 
are  indexed  from  0  to  N  rather  than  from  1  to  N ,  define 
(N+l) -vectors  for  each  of  the  input  channels  to  contain  their 
required  time  histories  as 


Xj(k)  =  [Xj(k) 


Xj (k-n) ] 


(A. 22a) 


and  embed  these  vectors  into  a  single  Q^(N+1) -data  vector 
given  by 


X  +  (k)  =  Cxt(k)T  ...  x!  )k)T] 

—  -l 


(A. 22b) 


214 


■i  - 


Define  a  Q^(N+1)  x  Qq  matrix  of  weights  as 


D*  = 


in 


^Q.l 

*1 


^0 


(A. 23a) 


where  the  (N+l) -vectors  d„  are  given  by 


dt .  =  [d. .(0)  ...  d. . (N)]T 
-13  i]  13 


(A. 23b) 


and  contain  the  coefficients  of  the  polynomial  d.. (z).  These 

J 

polynomials  can  be  combined  into  a  single  matrix  polynomial 
given  by 


J2+<z>  ■ 


dii<z> 


dQ.l(z) 


diQ  <z) 


dQ.Q  <Z> 

yix0 


(A. 23c) 


With  these  definitions,  the  N-th  order  MA  prediction  of  ^(k) 
is  given  by 


^(k)1  =  Xf(k)T  D+ 


(A. 24a) 
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(  , 


or  in  the  transform  domain, 


Y(z)T  =  X(z)T  D+(z) 


(A. 24b) 


Defining  a  prediction  error  vector  as 


rp  rn  A  cn 

en(k)x  *  £<k)  -  ^(k)1 


(A. 25a) 


and  setting  the  coefficients  in  _D+  to  minimize  the  trace 
of  the  prediction  error  covariance  matrix 


PQ  =  e{eQ(k)e0(k)  } 


(A. 25b) 


results  in  a  solution  given  by 


eCK+0O  X  +  Ck)T>  D+  =  £<X+(k)  y.(k)T}  CA.?6a) 


*  +  + 
xlxl 


'*  +  + 
xQiXl 


*  +  +  ~ 
X,  x^ 


*  +  + 

xn  xn 
^i  *<i 


*1*1 


~  + 

V1 


...  V  + 


xlyQ0 


...  r  , 
—  + 


xn.y 


(A. 26b) 
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Adopting  a  shorthand  notation  this  becomes 


R+  D+  =  r 


(A. 26c) 


Assume  a  relationship  between  the  components  of  the 
(n+l)-st  and  n-th  order  solutions  given  by 


,+  (n+l)  _ 

~ij 


H+(n) 
d .  . 


(n+1) 


(n+1) 


(A. 27) 


Embedding  the  vectors  v.  .^n+^  and  "the  coefficients 

1 J  3.  J 

into  matrices,  designated  (j^n+^  and  G^n+^\  and  solving 
for  Them  in  the  (n+l)-st  order  MA  problem  it  is  shown  in 
Appendix  D  that 


^(n+1)  _  _j"(n+l)  G<n+1) 


CA. 2  8a) 


G(n+1)  _  pCn+l)'1^  (n+1).^"+l)TD+<n)] 

—  xy 


(A. 28b) 


where 


-=-(n+l)  — rCn+1)  —(n+l) 

r  amd  Pv  are  matrices;  that  emerge 


from  the  backv  ’d  prediction  problem  in  a  multichannel  (n+l)-st 
order  autoregression  on  the  input  signal  vector  x(k) .  Again, 

_t  is  clear  that  the  multichannel  HA  solutions  given  by 
equations  (A. 28)  are  a  matrix  algebra  generalization  of 
equations  (2.23)  for  the  single  channel  HA  model. 
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To  relate  successive  order  MA  matrix  polynomials  to  one 
another,  equations  (A, 27)  and  (A. 28a)  can  be  used  to  write 


D+(Z><n+1>  =  D+(z)(r')+z'nCX-DCn>(z)]G<n+1)  (A.  29) 


where  the  second  tern.  on  the  right  hand  side  that  premul- 
tipliee  G  is  the  backward  prediction  error  matrix  polynomial 
from  the  autoregression  on  the  input  signal  x(k).  Pre- 

HI 

multiplying  by  X^z),  transposing,  and  transforming  into 
the  time  domain  results  in  the  multichannel  equivalent  of 
equation  (2.37) 


^(n+l)(k)  a  ^n\k)  +  -(n+l)(k) 


(A. 30) 


and  completes  the  derivation  of  the  recursive  in  order 
solutions  for  the  multichannel  MA  model. 


APPENDIX  B 


A  Key  Relationship  For  The  Multichannel  Levinson  Algorithm 

Equation  (A. 13)  is  a  key  relationship  in  the  development 
of  the  Levinson  algorithm  responsible  for  much  of  the 
algorithm's  simplicity.  Without  this  relationship,  more 
than  just  the  K  and  K  matrices  would  be  needed  to  obtain 
the  new  model  from  the  previous  model. 

In  equation  (A. 7),  consider  the  multiplication  of  the 
i-th  block  row  of  JR  by  the  j-th  block  column  of  JD  to 
form  r  . 

xi  j 


+  R  d.  .  *  r 
“xixQc^V]  ~xixj 


(B.l) 


In  particular,  consider  a  general  term  on  the  left  hand 
side  of  equation  (B.l)  in  detail. 


(B .  2 ) 


Define  upside  down  versions  of  the  d..  and  r  „ 

-13  "xix3 
as 


vectors 


(B . 3a) 


21C 


and 


£xixj 


Rx  x  (N) 

i  j 


Rx  x  C1) 
xixj 


(B , 3b) 


Using  these  permuted  vectors  in  equation  (B.l)  in  place  of 
the  d  and  r  vectors ,  the  relationship  is  still  satisfied  if 
the  R  matrices  are  permuted  as  well.  In  particular,  from 
(B.2)  it  is  evident  that 


.  + 


•••  Vm(N"1) 

i  m  x  m 


R  Cl-N)-  ...  R  CO) 
x  in  x  m 


dij(N) 


dijd> 


+  . . .  = 


V:(N> 


Rx  x  (1) 
xixj 


or  equivalently 


(B . 4a) 


R 


-Vi 


f ,  .  +  ...  +  R 


A  “  P« 


~xixQn  ixixj 


CB.4b) 


Embedding  the  f . .  and  p  vectors  into  matrices  designated 

—xixj  _ 

F  and  respectively  and  using  the  definition  of  JR  it 


follows  that 


R  F  =  /> 


(B .  5) 
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Defining  7. .  and  £  as  upside  down  versions  of  their 

xixj  —  — 

corresponding  vectors  in  JD  and  _T_  in  equation  (A, 11)  and 
embedding  them  in  matrices  designated  _F_  and  it  also 
follows  from  a  similar  development  that 

R  £  =  /£  (b.6) 

Equations  (B.5)  and  (B.6)  are  essential  to  the  develop¬ 
ment  of  the  Levinson  algorithm  and  are  made  possible  by  the 
Toeplita  structure  of  the  block  components  of  the  R  and 
R  matrices. 


APPENDIX  C 

The  Multichannel  Levinson  Algorithm  Solution 

For  AR  Modeling 

In  the  (n+l)-st  order  versions  of  equations  (A. 7)  and 
(A. 10),  the  component  matrices  that  make  up  R  and  It  can 
be  written  as  follows 


Sx'x'1’  - 
xixj 


p  (n) 
Sxixj 


x .x. C-N) 

j 


RX.X . (-1) 


txix,(N)-'-Rx.Xj'1)|  Rx . x . ( 0 ) 


xixj 


I  p 

I  £xix: 


(C.l) 


p (n) 

,  jxi 


x^Xj (0) 


(n+1) 

Ix.X. 

1  0 


(n+1) 

V3 


R(n)T 

“xixj 


I 

^-x.x. 

.L _ tS 


(C .  2) 


P 

-x.x. 


x^Xj ( 0 ) 
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■  --jfiife*  l2£a1L'JBk&ik  rS£sfc**yC.-*., 


Additionally , 


BWmibsb:-  . . •  -■-  ■”*—  * 


r(n+l) 

“xixj 


„(n> 


x^x ^ (n+1) 


(C .  3 ) 


With  these  matrices  and  vectors  written  in  this  partitioned 
form,  and  with  the  relationships  assumed  in  equations  (A. 15) 
between  the  n-th  and  <n+l)-st  order  solutions,  the  (n+l)-st 
order  modeling  equations  become: 

Forward  Model: 

£(n)  p(n)  +  R(n)  c  Cn)  +  ^(n)KCn+l)  s  j^Cn)  (C.4a) 

^Cn)T  J)(n)  +  ^h)T  £in)  +  ^(Q)^(n+l)  3  R^Cn+l) 

(C.4b) 


Backward  Model: 

|J(n)  £j(n)  +  R(n)T  ef(n)  +  /?Cr0  £C^+L)  s  J^(n) 

(C . 5a) 

^(n)T  g(n)  +  ^(n)T  £(n)  +  ^(0)^(n+l)  a  R^Cn+l)”1 

(C.5b) 
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Equations  (C.4a)  and  <C,5a)  contain  the  n-th  order  modeling 
equations  within  them  however,  and  therefore  can  be  written 
as 


J^(n)  £  (n)  _  _  n)  K(n+1) 


(C.6a) 


jj^(n)  £(n)  _  _  ^(n)  ^(n+1) 


(C.6b) 


and  applying  the  relationship  developed  in  Appendix  B 
(equations  (B.5)  and  (B.6))  yields 


^(n)  s  _  J'(n)  ^(n+1) 


(C.7a) 


^(n)  ■•  =  -  F(n)  ^Cn+i) 


(C.  7b) 


Thus  the  e  and  if  vectors  have  been  found  in  terms  of  the 
known  quanties  f  and  F  and  the  unknown  K  and  Ef  matrices. 
Substituting  equations  (C.7)  into  (C.4b)  and  (C.5b)  completes 
the  solution  resulting  in  expressions  for  the  K  and  K 
matrices  given  by 

K(n+1)  =  [R  <0)  -  T(n>TD(n)]'1[Rvv(n+l)-^<n,TD(n)] 

"  -*XX  n  1  "  ■ 

(C.8a) 
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^  ?i!f!,^,Wrll^ii[  ii  jjnf  t 


I-WWI^.I^.II,.'.  IJ  J,...ip,-. 


sasasa 


£(n+1)  =  [r  v<o)- r  (n)TD(n)3  1[r  v<n+i)T-/<?(n)'rD<n)] 

““XX  ■  " 1  ““XX  '  "  ■  1  “  “ 


(C. 8b) 

The  terms  on  the  right  hand  side  of  these  equations  that 
are  inverted  are  just  the  backward  and  forward  prediction 
error  covariance  matrices,  for  the  optimum  n-th  order  models 
given  by 


p<n)  =  r  (o)  -  r(n)  D<n) 

~a~  ““XX  ■  ■ _ "  — 


(C.9a) 


F(n)  -  r  (0)  -  r(n)TD(n) 

-  -xx  -  - 


(C.9b) 


Using  equations  (C.3),  (A. 15)  and  (C.7),  the  forward  pre¬ 
diction  error  covariance  matrix  for  the  optimum  (n+l)-st 
order  model  can  therefore  be  written  as 


p ( n+ 1 )  _  R  (0  r  (n)T  J)(n)+ r  (n)Tp  (n)K(n+l) 
—  — XX  _  1  ~  n_  "* 


-  R  (n+l)TK(n+1> 
-xx  - 


(C .10a) 


>T-r 


=  p<n)  +  cr(n)  FCn)-R  (n+l)T]K(n+1)  (C.  10b) 


-xx 


-<*» 


P(n)[x-p(n)  1[r  (n+l)T-/^(n)  D(n)]K(n+1)] 

r"r"  "“o  A  SB” 


(C.lOc) 
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(C.lOd) 


p(n+l)_  p(n)[j^(n+l)K(n+l)  -j 


and  following  a  similar  development  for  the  backward  pre¬ 
diction  error  covariance  matrix  results  in 


^(n+1)  _  _  K(n+l)^Cn+l)|j 


(C.ll) 


i 
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APPENDIX  D 

The  Multichannel  Levinson  Algorithm  Solution 
For  MA  Modeling 


First  note  that  because  of  the  definitions  of  the  x^(k) 

and  x^(k)  vectors  (indexed  from  0  to  n  and  1  to  n  respec- 

tively)  in  the  n-th  order  models,  the  matrices  R  +;+  in 

x  •  x . 

the  n-th  order  MA  problem  could  also  be  written  J  in 
terms  of  x^(k)  and  x^(k)  as  (assuming  stationarity ) 


p(n) 

-  +  + 
xixj 


«(n+l) 

"xixj 


(D.la) 


so  that 


R+(n)  a  r (n+1) 


(D. lb) 


(n+1) 


The  components  of  R  and  X 


(n+1) 


in  the  (n+l)-st 


order  MA  modeling  problem  can  be  written  in  partitioned  form 


R (n+1) 
-  +  + 

xixj 


R(n) 

£  +  + 

XfX. 


— <n+l)Tj 
E-x.x.  , 


(n+1) 


fix,x 


(D . 2a) 


Sc  .x.  (0) 

^  J 
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and 


E(?+1) 


xiyj 


xiyj 


V3(n+1) 


CD. 2b) 


Using  these  partitioned  forms,  and  the  relationship  in 
equation  (A. 27)  between  the  n-th  and  (n+l)-st  order  solutions, 
the  (n+l)=st  order  modeling  equations  become 

R+(n)  Dt<n)  *  R  +  <n>  l(n+1)  ♦  ^(n+iyntl)=i+(n> 

CD. 3a) 

^(n+l)TD+(n)  +  ^(nn)T  JCn+1)  +  *  (0)e<n+l> 

1  "  - 1  —  "XX  ii,r 

s  *Xy(n+1)  CD. 3b.) 

Equation  (D.3a)  contains  within  it,  the  modeling  equation  for 
the  n-th  order  MA  model  and  using  equation  (D.lb)  can  be 
rewritten  as 


j^(n+l)  fj  (n+1)  _  _  ^Cn+D^Cn+l) 


(D .  4 ) 


A  comparison  of  this  result  with  equation  CC.6a)  shows  that 


h  (n+1)  _  __  p"(n+l)g(n+l) 
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where  JT^n+3J  perJnu-ted  version  of  the  backward 

solution  in  an  (n+l)~st  order  AR  model  of  the  input  signal 
vector  x(k) .  Furthermore,  substituting  equation  (D.5)  into 
(D.3b)  results  in  a  solution  for  G^n+^  given  by 


,<n+l)  „ 


=  CRXXC°)  -  £ 


/<,(n+l)T  p  (n+l)  n*1 


[K  <n*l)-/£(n+1)  DHn)] 
-xy 


(D.  6) 


Since 


^(n+1)^  p  (n+1)  _  j*  (n+l)^^Cn+l) 


it  follows  that  the  inverted  term  on  the  right  hand  side 
of  equation  (D.6)  is  just  the  backward  prediction  error 
covariance  matrix  for  the  optimum  (n+l)-st  order  AR  model 
on  the  input  signal  vector  x(k). 
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**  * 


APPENDIX  E 


Prony's  Method  For  ARMA  Modeling 


Prony’s  method  [Refs.  8,  52  and  56]  obtains  a  zero  pole 
model  for  a  system  by  match  ,.np  the  impulse  resporfses  of  the 
system  and  model  over  the  first  M+N+l  sample  intervals  where 
M  and  N  are  the  orders  of  the  model  transfer  function  num¬ 
erator  and  denominator  polynomials.  Assume  that  a  signal 
y(k)  is  available  that  represents  the  impulse  response  of 
a  causal  system  and  that  a  rational  transfer  function  model 
for  this  systems  is  desired.  Using;  a  to  denote  the 
model  output  and  u(k)  to  denote  the  input  to  both  the  system 
and  model,  the  model  transfer  function  is  given  by 


HCz)  = 


A  £  a(n)z~ 

Y(z)  _  n=0 

uTzT - N - 

1+  Z  b(n)  z" 
n=l 


(E.l) 


For  a  unit  sample  input  it  follows  that 


B(z)  Y(z)  =  A(z) 


(E.  2) 


Equating  like  powers  of  z  in  this  relationship  results  in 


ia(n)  \  0  <  n.  j<  M 


b(i)  y(n-i)  = 


(E .  3) 


else 


where  b(0)=l. 
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Equating  y(k)  and  y(k)  over  the  interval  0  _<  k  _<  M+N  produces 
a  set  of  M+N+l  equations  which  can  be  expressed  in  matrix 
form  as 


(E . 4a) 


•«  * 
s  *■ 


or  adopting  a  shorthand  notation 


“  + 

*1 

iT 

1 

~  f 
a 

*2 

S2 

b 

0 

Assuming  that  the  NxN  matrix  Y2. 


b  -  -y;1  y2 

a+  =  4  -  Si  I'ihi 


(E. 4b) 

is  not  singular  if  follows  that 

(E.5a) 

(E,Sb) 
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The  original  Prony  method  goes  on  to  form  a  partial 
fraction  expansion  and  inverse  transformation  on  the  model 
transfer  function  H(c)  resulting  in  a  model  for  the  impulse 
response  of  the  system  given  as  a  sum  of  complex  exponentials. 
This  in  unnecessary  here  however,  since  a  rational  transfer 
function  model  is  the  form  sought. 

Prcmy’s  method  inherently  assumes  that  matching  a 
sufficient  portion  of  the  impulse  response  of  the  system 
results  in  an  acaurate  model  but  this  is  not  necessarily 
the  case  unless  the  impulse  response  damps  out  quickly  or 
unless  the  system  can  be  represented  exact ly  by  a  low  order 
model.  Otherwise  a  very  high  order  model  may  be  required 
to  obtain  sufficient  accuracy.  Other  difficulties  asso¬ 
ciated  with  this  technique  arc: 

1. )  The  system  impulse  response  must  be  available; 

2. )  There  ar*  no  built  in  mechanisms  to  test  for  or 

ensure  stability  of  the  model; 

?.)  There  is  no  averaging  of  noise  in  the  data; 

u , )  Only  a  raimvll  portion  of  the  available  data  points 
(M+N+l)  are  actually  used. 

The ilitf  difficult iuo  can  be  partially  overcome  by  modifying 
the  procedure  tc  obtain  an  approximate  match  of  the  system 
and  model  responses  over  their  entire  duration  rather  than 
an  enact  mat oh  over  the  first  M+N+l  points.  Consider 
equations  v,E.4)  modified  to  include  the  entire  signal  y(k) 
for  0  <  k  <  », 
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0 


1 


y  ( 0 ) 

yd> 


v(  0 ) 


* 


y(M)  |  y('M-l) 

. f- . . 

y(M+l)  |  y(M)  .  . 


(E.6a) 


Adopting  a  shorthand  notation  this  becomes 


*L  !  *1 

1 

M 

a 

. f . 

... 

2 

t 

b 

0 

*3  j  —3 

(E. 6b) 


This  yields  two  equations 

*i  +  h.  ± 

Z3  +  y3  b  =  o 


(E. 7a) 

(E,7b) 


but  with;y3,  Y3 
equation  (E.7b) 


and  £  having  an  infinite  number  of  rows, 
will  in  general  have  no  solution.  In  practice 


only  a  finite  portion  of  the  system  impulse  response  y(k) 
can  be  considered  but  equation  (E.7b)  will  still  be  incon¬ 
sistent  in  general.  A  least  squares  estimate  of  b  can  how- 

T 

ever,  be  obtained  by  minimizing  e  e  where 


e  =  Y3  b  -  (-Zs) 


(E. 8a) 


resulting  in 


b 


«3£3> 


-1 


(E . 8b) 


which  in  turn  can  be  used  in  equation  (E.7a)  to  find  a  . 
This  approximate  version  of  Prony's  method  is  the  one  most 
commonly  applied. 
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APPENDIX  F 


Parabolic  Surfaces  In  n-Dimensions 

Multi-dimensional  parabolic  surfaces  are  described  by 
an  equation  of  the  form 

y  =  xTAx  -  2  xT  b  +  c  (F.l) 


where : 

y  is  the  independent  variable1, 
x  is  a  vector  of  dependent  variables j 

A  is  a  symmetric  constant  matrix*, 

b  is  a  constant  vector*, 

c  is  a  constant. 

(x,  b  and  c  can  also  be  considered  as  matricies  with  the 
trace  of  the  right  hand  side  set  equal  to  the  independent 
variable  but  the  problem  remains  essentially  unchanged.) 
Completing  the  square  for  nonsingular  A  this  becomes 

y  —  ( x  —  A-1b)T  A(x  -  A'Xb)  +  c  -  bTA_1b  (F.2) 

so  that  for  positive  definite  A  it  is  clear  that  the  minimum 

value  of  y  is  obtained  for  x  =  A-1b  and  this  minimum  is 
T  -1 

c  -  b  A  b.  It  is  also  clear  that  nonzero  values  of  b  and  c 
simply  raise  and  lower  the  surface  and  move  the  minimum 
point  away  from  x  =  £,  The  shape  of  the  surface  (its 
relative  concavity  or  flatness)  is  determined  by  the  matrix 


235 


A  in  the  quadratic  term  of  equation  (F.l).  Therefore,  to 
study  the  shape  of  this  surface,  consider  the  simpler  pro¬ 
blem  with  b  and  c  set  to  zero. 


y  =  x  A  x 


(F.3) 


One  way  to  examine  the  relative  flatness  or  concavity  is  to 
look  at  the  locus  of  points  on  the  surface  for  constant 
values  of  y*,  in  particular  when  y=l.  Recognize,  that  A  can 
be  rewritten  as  Q  AQ  where  A  is  a  diagonal  matrix  of 
eigenvalues  and  ^  is  a  matrix  whose  columns  are  the  unit 
length  eigenvectors  of  A.  Now  (F.3)  becomes 


T  T 

xa2  a  a  x  =  1 


(F.4) 


and  introducing  a  new  set  of  variables  (which  are 

simply  a  rotation  of  the  variables  in  x) ,  equation  (F.4) 
reduces  to 


X,w?  +  ...  +  X  w^  =  1 
11  n  n 


(F.  5 ) 


This  equation  describes  an  ellipsoid  in  n  dimensions  whose 
axes  half  lengths  are  given  by  1/AT  for  1  <  i  <  n.  This 
follows  from  letting  all  but  one  of  the  w’s  equal  to  zero 
and  solving  for  the  nonzero  variable  so  that  one  point  on  the 
surface  is  for  example  w^  =  1/ /X^  with  w2  =  . . ,  =  wn  =  0 , 
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l&ffpwww’vi ">i.i _ _ ,,_ _ _  .._  „t  ,  -„r _ J _ , _ r^,  ^—t--... _ , _ ^_rf—  ,^Tt-  . . . _  _ _ _  ,  _ _ _ _ ___ _ _ _  -  . . 

This  point  is  just  a  multiple  of  the  first  eigenvector  of 
A  so  that  in  general,  the  axes  of  the  ellipsoid  point  in 
the  direction  of  the  eigenvectors  of  A  with  half  lengths 
given  by  the  recriprocal  of  the  square  root  of  the  eigen¬ 
values  . 


V  > 
*  r 


\ 
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APPENDIX  G 

Multichannel  ARMA  Modeling 

Consider  a  system  with  Qq  output  signals  [y^(k)...yq  (^)] 

and  Q.  input  signals  Cu. (k)  ...  un  Ck)].  The  multichannel 
1  l 

ARMA  analysis  model  forms  an  estimate  of  the  present  value 
of  each  output  as  a  weighted  combination  of  past  values  of 
all  output  signals  and  past  and  present  values  of  all  input 
signals . 


!0  N 


y  (k) 


n 


*  2  b.  (i)  y.Ck-i) 

3*1  i*l  J  3 


Qi  N 


-  Z  ajn(i)  u.(k-l) 


(G.l) 


Define  data  vectors  for  all  the  input  and  output  channels  as 


y  (k)  *  Cy_(k-1)  ...  y  (k-N)]' 
tn  n  J  n 


(G.2a) 


u*(k)  -  Cu  (k)  ...  u  (k-N)]T 
— n  n  n 


(G.2b) 


and  embed  them  into  Q^N  and  Q^(N+1)  vectors  given  by 


V  »  tj^Ck) 


*Q, 


Ck)T]' 


( G . 3a) 


4  h 
4  *> 


U+  =  Cu*(k)T 


ut  (k)T] 
“^i 


(G. 3b) 
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and  setting. the  model  coefficients  to  minimize  the  trace  of 
the  prediction  error  covariance  matrix  results  in  a  solu¬ 
tion  given  by 


5yv  E  + 

YU 


R  +  £  +  + 

U  Y  U  U 


(G.7) 


In  the  transform  domain,  the  prediction  error  model  is  re¬ 
presented  by 


E Q ( z ) T  =  Y(z)T  Cl-B(z) ]-U( z)T  A(z) 


(G.  8) 


where  E,  i  and  U  are  the  transforms  of  the  error,  output 
and  input  vectors  and  the  coefficients  of  the  i,j-th  elements 
of  the  matrix  polynomials  B(z)  and  A(z)  are  the  elements  of 
the  vectors  b^  and  at  ^ .  The  multichannel  ARMA  synthesis 
model  is  then  given  by 


YT(z)  =  U(z)T  A(z)Cl-BCz)]"1 


(G .  9) 


with  the  matrix  polynomial  fraction  serving  as  the  generali¬ 
zation  of  the  zero  pole  transfer  function. 


APPENDIX  H 


Delay  Free  Loops 

To  develop  equation  (4.24)  guaranteeing  the  absence  of 
delay  free  loops  in  the  system  of  Figure  4.8,  consider  the 
equation  for  y^(k) . 

2^00  =  FCXjjCk)]  (H.l) 

Since  F[ . ]  is  a  raemoryless  nonlinear  function,  proving  that 

Xjj(k)  is  not  a  function  of  ;^(k)  is  equivalent  to  proving 

that  y^(k)  is  not  a  function  of  itself,  and  therefore  no 

delay  free  loops  exist.  From  equations  (4.17b)  and  (4.17d) 

with  u(k)=0  it  follows  that 

— "  * 

£N(z)  =  I2  -<z)  -1  ^NCz)  (H,2) 

and  the  coefficient  of  y^  at  time  k  on  the  right  hand  side 
is  given  by 

4 

a  =  Lim  £2  T(z)  £  (H.3) 

Z-Voo 

A  nonzero  a.^  indicates  a  dependence  of  xjij^(k)  upon  y^j(k) 
and  clearly  therefore  all  the  main  diagonal  elements  of  a 
must  be  zero  to  avoid  delay  free  loops.  These  elements  are 
the  terms  of  the  (Q-<l)-st  principal  minors  of  a.  While  this 
is  a  necessary  condition  it  is  not  sufficient  to  avoid  delay 
free  loops  since  loops  may  exist  through  two  or  more  signals. 


The  condition  that  a.,  a..  =  0  for  all  i,j  such  that 

^3  3 

1  <  i,j  <  Q  and  i?£ j  t  ensures  that  no  delay  free  loop  exist 
through  2  signals  and  is  equivalent  to  requiring  all  terms 
of  the  (Q-2)-nd  principal  minors  of  a  are  zero.  A  term  of 
a  determinant  [Ref.  63]  is  defined  as  the  product  of  elements 
of  the  matrix  taken  one  from  each  row  and  one  from  each 
column  and  the  determinent  of  the  matrix  is  the  sum  of  all 
possible  terms.  It  is  clear  therefore  that  every  term 
must  contain  a  cycle  such  as  a^.ou^.  *  **aii  anc*  must  there¬ 
fore  be  zero.  Since  the  determinant  consists  of  every 
possible  term,  requiring  that  all  terms  of  the  determinants 
of  the  (Q-i)-th  principal  minors  are  zero  ensures  that  no 
delay  free  loops  exist  through  any  combination  of  i  signals. 
Examining  all  terms  of  the  determinant  of  a  and  all  its  ? 
principal  minors  ensures  that  all  possible  loops  through 
the  Q  signals  in  x^(k)  are  examined.  If  any  delay  free  loop 
exists ,  then  at  least  one  of  the  terms  of  one  of  the  deter¬ 
minants  will  be  nonzero. 


APPENDIX  I 

Nonlinear  ARMA  Modeling  Examples 

The  determination  of  memory  requirements  for  the  non¬ 
linear  ARMA  model  as  well  as  its  applicability  to  systems 
consisting  of  interconnections  of  linear  and  memoryless  non¬ 
linear  subsystems  is  illustrated  here  using  two  examples. 
First  a  cascade  of  linear  and  nonlinear  subsystems  is  con¬ 
sidered.  Then  a  real  world  example  is  considered  and  a 
nonlinear  ARMA  model  is  proposed  for  the  tracking  behavior 
of  a  phase  locked  loop. 

A.  A  CASCADE  OF  LINEAR  AND  NONLINEAR  SUBSYSTEMS 

Consider  the  system  shown  in  Figure  1.1  where  the  signals 
u(k)  and  yLZ(k)  are  observed.  In  terms  of  the  topology  of 
Figure  4.8,  seven  signals  can  be  identified  (x^)  xL2*  yLl, 
yL2?  XN1*  yNl  and  however  for  convenience  three  of  the 
seven  equations  in  (4.26)  which  specify 


x^^Ck)  =  u(k) 

XL2<k)  S  yNl(k) 
XNl^k^  5  yLl(k) 


(I. la) 

(I. lb) 

(I.lc) 


will  not  be  explicitly  written.  In  this  case,  equation 
(4,26)  becomes 


Figure  1.1.  A  Nonlinear  System 


(1.2) 

where  the  finite  memory  representations  of  T-^(z)  and  T2(z) 
have  been  used.  The  objective  now  is  to  eliminate  a2(k)* 
from  the  upper  right  partition.  Using  the  equations  of  rows 
three  and  four,  the  first  row  can  be  rewritten  as 

yL2(k)=b2(k)*  yL2(k)+a2(k)*F1Ca1^)*^(kHb1(k)*yL;L(k)] 

(1,3) 

Notationally  it  is  difficult  to  write  equation  (1,3)  in  the 
operator  matrix  form  of  equation  (1.2)  because  of  the  non¬ 
linear  function  however,  the  following  representation  is 
adopted. 
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No  further  reductions  are  possible  to  make  the  upper  right 
partition  a  null  matrix  leading  to  the  conclusion  that  for 
a  finite  memory  nonlinear  ARMA  model  to  be  appropriate, 
either  b^(k)  must  be  zero  (the  first  linear  system  must  be 
nonrecursive)  or  y^(k)  must  also  be  observed,  Alternately, 
if  b^(k)  ia  nonzero,  a»*  infinite  memory  representation  can 
be  used  for  the  first  linear  system  by  replacing  a,(k)*(.) 
with  h^<k)rt< , >  and  b^(k)A(.)  with  zero  in  equation  (1.4) 
indicating  that  an  infinite  memory  nonlinear  ARMA  model  is 
appropriate  when  only  u(k)  and  y^2(k)  are  observed. 

U.  A  NONLINEAR  ARMA  MODEL  FOR  A  PHASE  LOCKED  LOOP 

A  continuous  time  model  for  the  tracking  behavior  of  a 

» 

nase  looked  loop  [Ref,  S5]  is  shown  in  Figure  1.2  where: 
0^(t)  is  the  phase  of  the  incoming  signal 
l^t)  i#  the  estimate  of  the  phase  of  the  incoming  signal 
e(t)  is  the  phase  error  signal 
F < a )  is  the  transfer  function  of  the  loop  filter 
and  K2  are  constants 


Figure  X.2.  A  nonlinear  model  for  the  tracking 
behavior  of  a  phase-locked  loop. 


The  model  is  nonlinear  because  of  the  sin  function  in  the 
loop.  Often  the  assumption  is  made  that  e(t)<<ir/2  so 
that  sin  eCt)  "  e(t)  in  which  case  a  linearized  model  is 
obtained  as 


0o(s) 

e^TsT  = 


K1K2J(s) 


(1.5) 


A  nonlinear  ARMA  model  for  the  system  can  be  obtained  by 
first  discretizing  the  model  of  Figure  1.2  as  shown  in 
Figure  1.3  where  F(z)  represents  the  discrete  loop  filter. 


Figure  1.3.  A  discrete  version  of  the  phase 
locked  loop  model. 


2 

The  integration  has  been  approximated  as  - r  ,  (It  is 

1-z 

necessary  to  use  this  Euler  forward  approximation  for 
integration  rather  than  one  such  as  the  trapezoid  rule  to 
avoid  a  delay  free  loop).  Defining  a  single  linear  system 
aq  the  cascade  of  FCz)  and  the  discrete  integration 

KiS  7^4  F(z)  "  rarer 

1-Z 
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Equation  (4.26)  for  the  phase  locked  loop  becomes 


rx 

CM 

L-fi- 

bx(k)*  0  |  ax(k)*  0 

l — 

X 

CM 

e1(k) 

0  1  1  0  0 

1 

e1Ck) 

yN<k) 

o 

o 

— 

o 

10 

H- 

K 

yN^) 

e(k) 

-1  1  '  0  0 

e(k) 

(1.7) 


where  it  is  assumed  that  0^(k)  and  8g(k)  are  observed.  Using 
the  relationships  specified  by  rows  3  and  4 ,  this  can  be 
written  as 


02(k) 

01(k) 

r- 

b, (k)*  0  l  0  0 

1  1 

0  1  I  0  0 

.  .  .  ,  1 

02oo 

S-^k) 

yN(k> 

0  0^0  sin(.) 

yN(k) 

e(k) 

—  — 

j 

-1  10  0 

— 

e(k) 

—  « 

a]_(k)*  sin[-( , )  (.)]  |  0  0 

02(k) 

0  0^00 

6,  (k) 

0  0  1  0  0 

| 

yNOc) 

0  0  j  0 

_e(k)  __ 

(1.8) 


from  which  it  is  clear  that  a  finite  memory  nonlinear  ARMA 
model  is  appropriate.  The  model  can  be  obtained  from  the 
first  row  in  equation  1.8  by  substituting  a  series  expansion 
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the  sin  function  and  truncating  it  at  the  degree  desared 


resulting  in 


+  a,  (k)  * 


9,(k)=b.(k)'»e2(k)  +  a^k)*  2  •a®T5T(e2(1<;)"9l(k))21” 

Z  u  ^  ra=  u 

(1.9) 

where  2M+1  is  the  degree  of  the  series  approximation  to 
sin( . )  •  (Note  that  lim  A]_(z)  =  lim  B^z)  =  0  so  that  the 
right  hand  side  of  equation  (1.9)  involves  only  past  values 
of  the  output  8jOO.)  An  infinite  memory  Volterra  series 
model  for  this  system  has  been  considered  by  Van  Trees 

[Ref.  55], 


APPENDIX  J 

Model  Simulation  Program  Listings 

This  appendix  provides  a  listing  of  the  fortran  model 
•simulation  programs  used  in  the  experimental  study  of  the 
lattice  characteristics.  Included  are  the  main  programs 
for  the  batch  processing  ARMA  lattice,  the  adaptive  ARMA 
lattice  and  the  brute  force  model  solution.  Each  main 
program  is  followed  by  a  collection  of  subroutines  used  only 
by  that  specific  main  program.  Then  a  collection  of  common 
subroutines  called  from  two  or  more  locations  in  the  batch 
lattice,  adaptive  lattice  or  brute  force  method  is  listed. 
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